2023-10-12T18:30:22.212 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-12T18:30:22.213 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584 --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} --owner scheduled_yuriw@teuthology --name yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi --verbose -- /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584/orig.config.yaml 2023-10-12T18:30:22.250 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-12T18:30:22.337 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584 branch: wip-yuri2-testing-2023-10-11-0853-pacific description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7423584' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri2-testing-2023-10-11-0853-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) sha1: 84698e743199450ee5f6969d22ebe6bed3a68e4a wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 84698e743199450ee5f6969d22ebe6bed3a68e4a rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-qa-63004-pacific sha1: e5d3c6c93855bbe531cbf7186fc3967731bde287 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 4772 sha1: 84698e743199450ee5f6969d22ebe6bed3a68e4a sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-qa-63004-pacific suite_path: /home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: e5d3c6c93855bbe531cbf7186fc3967731bde287 targets: smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1m8XjWbqD1iXBJEhpX/v8gnKr8KzzzsWDDpPq6Zl+EF15uFoyzlAjob/fRe4yJl3EC86XUqN4NQNwEQD1PzoY= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKwWJMgAdGT2iiMDleMD67QKtvCoO64aQexf5vcn57O0419KVr8ejNnxld6r3GrUe8l2tWI+ssJ7+UJv7eJPIME= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-12_17:57:07 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-12T18:30:22.338 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa; will attempt to use it 2023-10-12T18:30:22.338 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks 2023-10-12T18:30:22.339 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-12T18:30:22.341 INFO:teuthology.task.internal:Checking packages... 2023-10-12T18:30:22.365 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '84698e743199450ee5f6969d22ebe6bed3a68e4a' 2023-10-12T18:30:22.366 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-12T18:30:22.366 INFO:teuthology.packaging:ref: None 2023-10-12T18:30:22.367 INFO:teuthology.packaging:tag: None 2023-10-12T18:30:22.367 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-10-11-0853-pacific 2023-10-12T18:30:22.367 INFO:teuthology.packaging:sha1: 84698e743199450ee5f6969d22ebe6bed3a68e4a 2023-10-12T18:30:22.367 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2023-10-11-0853-pacific 2023-10-12T18:30:22.599 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-75.g84698e74 2023-10-12T18:30:22.599 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-12T18:30:22.606 INFO:teuthology.task.internal:no buildpackages task found 2023-10-12T18:30:22.606 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-12T18:30:22.627 INFO:teuthology.task.internal:Saving configuration 2023-10-12T18:30:22.642 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-12T18:30:22.648 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-12T18:30:22.675 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-12 18:25:17.519160', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1m8XjWbqD1iXBJEhpX/v8gnKr8KzzzsWDDpPq6Zl+EF15uFoyzlAjob/fRe4yJl3EC86XUqN4NQNwEQD1PzoY='} 2023-10-12T18:30:22.700 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-12 18:25:17.521010', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKwWJMgAdGT2iiMDleMD67QKtvCoO64aQexf5vcn57O0419KVr8ejNnxld6r3GrUe8l2tWI+ssJ7+UJv7eJPIME='} 2023-10-12T18:30:22.700 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-12T18:30:22.706 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-10-12T18:30:22.707 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-10-12T18:30:22.707 INFO:teuthology.run_tasks:Running task console_log... 2023-10-12T18:30:22.783 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbf379e1a60>, signals=[15]) 2023-10-12T18:30:22.784 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-12T18:30:22.790 INFO:teuthology.task.internal:Opening connections... 2023-10-12T18:30:22.790 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T18:30:22.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T18:30:22.861 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T18:30:22.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T18:30:22.927 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-12T18:30:22.933 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2023-10-12T18:30:22.951 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2023-10-12T18:30:22.952 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2023-10-12T18:30:23.008 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="Red Hat Enterprise Linux" 2023-10-12T18:30:23.008 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8.4 (Ootpa)" 2023-10-12T18:30:23.009 INFO:teuthology.orchestra.run.smithi170.stdout:ID="rhel" 2023-10-12T18:30:23.009 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="fedora" 2023-10-12T18:30:23.009 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8.4" 2023-10-12T18:30:23.009 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8" 2023-10-12T18:30:23.010 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-12T18:30:23.010 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2023-10-12T18:30:23.010 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-12T18:30:23.010 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://www.redhat.com/" 2023-10-12T18:30:23.010 INFO:teuthology.orchestra.run.smithi170.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-12T18:30:23.011 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-12T18:30:23.011 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:30:23.011 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-12T18:30:23.011 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-12T18:30:23.011 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-12T18:30:23.012 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-12T18:30:23.012 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2023-10-12T18:30:23.031 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2023-10-12T18:30:23.048 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2023-10-12T18:30:23.048 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2023-10-12T18:30:23.104 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Red Hat Enterprise Linux" 2023-10-12T18:30:23.104 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8.4 (Ootpa)" 2023-10-12T18:30:23.104 INFO:teuthology.orchestra.run.smithi187.stdout:ID="rhel" 2023-10-12T18:30:23.105 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="fedora" 2023-10-12T18:30:23.105 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8.4" 2023-10-12T18:30:23.105 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2023-10-12T18:30:23.105 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-12T18:30:23.105 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2023-10-12T18:30:23.105 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-12T18:30:23.106 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.redhat.com/" 2023-10-12T18:30:23.106 INFO:teuthology.orchestra.run.smithi187.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-12T18:30:23.106 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-12T18:30:23.106 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:30:23.106 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-12T18:30:23.107 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-12T18:30:23.107 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-12T18:30:23.107 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-12T18:30:23.107 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2023-10-12T18:30:23.131 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-12T18:30:23.143 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-12T18:30:23.149 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-12T18:30:23.150 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2023-10-12T18:30:23.153 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2023-10-12T18:30:23.168 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-12T18:30:23.174 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-12T18:30:23.174 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2023-10-12T18:30:23.209 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2023-10-12T18:30:23.222 INFO:teuthology.orchestra.run.smithi170.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-12T18:30:23.225 INFO:teuthology.orchestra.run.smithi187.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-12T18:30:23.226 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-12T18:30:23.278 INFO:teuthology.run_tasks:Running task kernel... 2023-10-12T18:30:23.292 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-12T18:30:23.293 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-12T18:30:23.294 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-12T18:30:23.294 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-12T18:30:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T18:30:23.310 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-10-12T18:30:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T18:30:23.312 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2023-10-12T18:30:23.366 INFO:teuthology.orchestra.run.smithi170.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-12T18:30:23.367 INFO:teuthology.task.kernel:Running kernel on smithi170: 4.18.0-305.25.1.el8_4.x86_64 2023-10-12T18:30:23.367 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2023-10-12T18:30:23.369 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-12T18:30:23.369 INFO:teuthology.task.kernel:Running kernel on smithi187: 4.18.0-305.25.1.el8_4.x86_64 2023-10-12T18:30:23.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2023-10-12T18:30:24.749 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:30:24.750 INFO:teuthology.orchestra.run.smithi170.stdout:Unable to read consumer identity 2023-10-12T18:30:24.824 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:30:24.824 INFO:teuthology.orchestra.run.smithi187.stdout:Unable to read consumer identity 2023-10-12T18:30:26.149 INFO:teuthology.orchestra.run.smithi170.stdout:Copr repo for python3-asyncssh owned by ceph 34 kB/s | 3.5 kB 00:00 2023-10-12T18:30:26.216 INFO:teuthology.orchestra.run.smithi187.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2023-10-12T18:30:26.766 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 27 MB/s | 16 MB 00:00 2023-10-12T18:30:27.101 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-10-12T18:30:30.115 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 365 kB/s | 24 kB 00:00 2023-10-12T18:30:30.364 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-10-12T18:30:30.838 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-12T18:30:30.838 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-12T18:30:30.838 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-12T18:30:30.997 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:30:30.998 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:30:30.999 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:30:31.034 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-12T18:30:31.035 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-12T18:30:31.035 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-12T18:30:31.110 DEBUG:teuthology.orchestra.run.smithi170:> echo no | sudo yum reinstall kernel || true 2023-10-12T18:30:31.179 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:30:31.180 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:30:31.181 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:30:31.279 DEBUG:teuthology.orchestra.run.smithi187:> echo no | sudo yum reinstall kernel || true 2023-10-12T18:30:31.455 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:30:31.455 INFO:teuthology.orchestra.run.smithi170.stdout:Unable to read consumer identity 2023-10-12T18:30:31.622 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:30:31.622 INFO:teuthology.orchestra.run.smithi187.stdout:Unable to read consumer identity 2023-10-12T18:30:31.748 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:01 ago on Thu 12 Oct 2023 06:30:30 PM UTC. 2023-10-12T18:30:31.874 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-12T18:30:31.874 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-12T18:30:31.875 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-12T18:30:31.909 INFO:teuthology.orchestra.run.smithi170.stderr:Error: No packages marked for reinstall. 2023-10-12T18:30:31.914 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:01 ago on Thu 12 Oct 2023 06:30:30 PM UTC. 2023-10-12T18:30:31.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum reinstall -y kernel || true 2023-10-12T18:30:32.055 INFO:teuthology.orchestra.run.smithi187.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-12T18:30:32.056 INFO:teuthology.orchestra.run.smithi187.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-12T18:30:32.056 INFO:teuthology.orchestra.run.smithi187.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-12T18:30:32.107 INFO:teuthology.orchestra.run.smithi187.stderr:Error: No packages marked for reinstall. 2023-10-12T18:30:32.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum reinstall -y kernel || true 2023-10-12T18:30:32.307 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:30:32.308 INFO:teuthology.orchestra.run.smithi170.stdout:Unable to read consumer identity 2023-10-12T18:30:32.500 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:30:32.501 INFO:teuthology.orchestra.run.smithi187.stdout:Unable to read consumer identity 2023-10-12T18:30:32.593 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Oct 2023 06:30:30 PM UTC. 2023-10-12T18:30:32.757 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-12T18:30:32.757 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-12T18:30:32.757 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-12T18:30:32.774 INFO:teuthology.orchestra.run.smithi170.stderr:Error: No packages marked for reinstall. 2023-10-12T18:30:32.789 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Oct 2023 06:30:30 PM UTC. 2023-10-12T18:30:32.824 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T18:30:32.935 INFO:teuthology.orchestra.run.smithi187.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-12T18:30:32.936 INFO:teuthology.orchestra.run.smithi187.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-12T18:30:32.936 INFO:teuthology.orchestra.run.smithi187.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-12T18:30:32.949 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:32.949 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:32.950 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:32.950 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-12T18:30:32.950 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-12T18:30:32.950 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-10-12T18:30:32.951 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2023-10-12T18:30:32.951 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T18:30:32.953 INFO:teuthology.orchestra.run.smithi187.stderr:Error: No packages marked for reinstall. 2023-10-12T18:30:32.999 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T18:30:33.079 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:33.079 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:33.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2023-10-12T18:30:33.106 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2023-10-12T18:30:33.106 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2023-10-12T18:30:33.107 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.02 2023-10-12T18:30:33.107 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 99.el8_4.1 2023-10-12T18:30:33.107 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2023-10-12T18:30:33.107 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-12T18:30:33.107 INFO:teuthology.orchestra.run.smithi170.stdout:Group : System Environment/Base 2023-10-12T18:30:33.108 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 9264631 2023-10-12T18:30:33.108 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2023-10-12T18:30:33.108 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-12T18:30:33.108 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-12T18:30:33.108 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-12T18:30:33.109 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-12T18:30:33.109 INFO:teuthology.orchestra.run.smithi170.stdout:Relocations : (not relocatable) 2023-10-12T18:30:33.109 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : Red Hat, Inc. 2023-10-12T18:30:33.109 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : Red Hat, Inc. 2023-10-12T18:30:33.109 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-12T18:30:33.110 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2023-10-12T18:30:33.110 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2023-10-12T18:30:33.110 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:30:33.110 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-12T18:30:33.110 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-12T18:30:33.111 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-12T18:30:33.111 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2023-10-12T18:30:33.111 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:30:33.111 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2023-10-12T18:30:33.112 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-12T18:30:33.112 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:33.113 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-12T18:30:33.125 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:33.126 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:33.126 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:33.126 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-12T18:30:33.127 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-12T18:30:33.127 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-10-12T18:30:33.127 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2023-10-12T18:30:33.127 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T18:30:33.253 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:33.254 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:33.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2023-10-12T18:30:33.281 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2023-10-12T18:30:33.281 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2023-10-12T18:30:33.281 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.02 2023-10-12T18:30:33.282 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 99.el8_4.1 2023-10-12T18:30:33.282 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2023-10-12T18:30:33.282 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-12T18:30:33.282 INFO:teuthology.orchestra.run.smithi187.stdout:Group : System Environment/Base 2023-10-12T18:30:33.282 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 9264631 2023-10-12T18:30:33.283 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2023-10-12T18:30:33.283 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-12T18:30:33.283 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-12T18:30:33.283 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-12T18:30:33.283 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-12T18:30:33.284 INFO:teuthology.orchestra.run.smithi187.stdout:Relocations : (not relocatable) 2023-10-12T18:30:33.284 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : Red Hat, Inc. 2023-10-12T18:30:33.284 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : Red Hat, Inc. 2023-10-12T18:30:33.284 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-12T18:30:33.284 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2023-10-12T18:30:33.284 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2023-10-12T18:30:33.285 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:30:33.285 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-12T18:30:33.285 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-12T18:30:33.285 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-12T18:30:33.285 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2023-10-12T18:30:33.285 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:30:33.286 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2023-10-12T18:30:33.287 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-12T18:30:33.287 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:33.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-12T18:30:34.388 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2023-10-12T18:30:34.702 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2023-10-12T18:30:36.240 INFO:teuthology.orchestra.run.smithi170.stderr:done 2023-10-12T18:30:36.241 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-10-12T18:30:36.296 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.OpdMq5A3xB 2023-10-12T18:30:36.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OpdMq5A3xB 2023-10-12T18:30:36.376 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.OpdMq5A3xB 2023-10-12T18:30:36.543 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.OpdMq5A3xB is 6KB 2023-10-12T18:30:36.592 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.OpdMq5A3xB 2023-10-12T18:30:36.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2023-10-12T18:30:36.673 INFO:teuthology.orchestra.run.smithi170.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-12T18:30:36.674 INFO:teuthology.orchestra.run.smithi170.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-12T18:30:36.674 INFO:teuthology.orchestra.run.smithi170.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-12T18:30:36.674 INFO:teuthology.orchestra.run.smithi170.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-12T18:30:36.675 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:36.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2023-10-12T18:30:36.859 INFO:teuthology.orchestra.run.smithi187.stderr:done 2023-10-12T18:30:36.861 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2023-10-12T18:30:36.875 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.EqfN8WbFHu 2023-10-12T18:30:36.875 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EqfN8WbFHu 2023-10-12T18:30:36.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.EqfN8WbFHu 2023-10-12T18:30:37.100 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.EqfN8WbFHu is 6KB 2023-10-12T18:30:37.149 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.EqfN8WbFHu 2023-10-12T18:30:37.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2023-10-12T18:30:37.230 INFO:teuthology.orchestra.run.smithi187.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-12T18:30:37.230 INFO:teuthology.orchestra.run.smithi187.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-12T18:30:37.230 INFO:teuthology.orchestra.run.smithi187.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-12T18:30:37.230 INFO:teuthology.orchestra.run.smithi187.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-12T18:30:37.231 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-12T18:30:37.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2023-10-12T18:31:06.874 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-10-12T18:31:06.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-12T18:31:06.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T18:31:07.384 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-10-12T18:31:07.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-12T18:31:07.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T18:32:06.935 DEBUG:teuthology.orchestra.remote:timed out 2023-10-12T18:32:07.388 DEBUG:teuthology.orchestra.remote:timed out 2023-10-12T18:32:15.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-12T18:32:15.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T18:32:16.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-12T18:32:16.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T18:32:16.397 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-10-12T18:32:16.663 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-12T18:32:16.664 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-12T18:32:16.664 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-10-12T18:32:16.717 INFO:teuthology.orchestra.run.smithi170.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-12T18:32:16.718 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:32:16.718 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-12T18:32:16.718 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-12T18:32:16.881 DEBUG:teuthology.orchestra.run.smithi187:> true 2023-10-12T18:32:17.165 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-12T18:32:17.166 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-12T18:32:17.166 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2023-10-12T18:32:17.220 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-12T18:32:17.221 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-12T18:32:17.221 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-12T18:32:17.222 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-12T18:32:17.719 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-10-12T18:32:17.720 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-12T18:32:17.856 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2023-10-12T18:32:17.870 DEBUG:teuthology.parallel:result is None 2023-10-12T18:32:18.223 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-10-12T18:32:18.223 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-12T18:32:18.366 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2023-10-12T18:32:18.386 DEBUG:teuthology.parallel:result is None 2023-10-12T18:32:18.386 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-12T18:32:18.395 INFO:teuthology.task.internal:Creating test directory... 2023-10-12T18:32:18.396 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-12T18:32:18.398 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-12T18:32:18.414 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-12T18:32:18.424 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-12T18:32:18.435 INFO:teuthology.task.internal:Creating archive directory... 2023-10-12T18:32:18.435 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-12T18:32:18.458 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-12T18:32:18.495 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-12T18:32:18.504 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-12T18:32:18.505 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-12T18:32:18.528 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-12T18:32:18.552 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T18:32:18.562 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T18:32:18.596 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T18:32:18.608 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T18:32:18.609 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-12T18:32:18.623 INFO:teuthology.task.internal:Configuring sudo... 2023-10-12T18:32:18.623 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-12T18:32:18.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-12T18:32:18.651 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-12T18:32:18.661 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-12T18:32:18.662 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-12T18:32:18.696 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-12T18:32:18.710 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T18:32:18.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T18:32:18.938 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T18:32:18.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T18:32:19.024 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:32:19.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-12T18:32:19.092 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T18:32:19.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T18:32:19.223 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T18:32:19.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T18:32:19.306 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:32:19.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-12T18:32:19.394 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2023-10-12T18:32:19.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2023-10-12T18:32:19.463 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T18:32:19.510 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T18:32:20.010 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-12T18:32:20.019 INFO:teuthology.task.internal:Starting timer... 2023-10-12T18:32:20.019 INFO:teuthology.run_tasks:Running task pcp... 2023-10-12T18:32:20.041 INFO:teuthology.run_tasks:Running task selinux... 2023-10-12T18:32:20.052 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-12T18:32:20.053 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2023-10-12T18:32:20.791 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: [ OK ] 2023-10-12T18:32:20.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2023-10-12T18:32:20.891 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: [ OK ] 2023-10-12T18:32:20.893 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-12T18:32:20.893 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2023-10-12T18:32:21.112 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2023-10-12T18:32:21.112 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2023-10-12T18:32:21.138 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2023-10-12T18:32:21.138 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi170.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2023-10-12T18:32:21.139 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T18:32:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T18:32:21.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T18:32:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T18:32:21.338 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-12T18:32:21.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2023-10-12T18:32:21.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2023-10-12T18:32:21.777 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-12T18:32:21.789 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-12T18:32:21.799 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-12T18:32:21.927 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-12T18:32:21.943 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-12T18:32:21.945 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi170.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-12T18:40:37.336 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi170.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2023-10-12T18:40:37.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-12T18:40:37.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T18:40:37.410 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-10-12T18:40:37.473 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-12T18:40:37.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-12T18:40:37.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T18:40:37.541 DEBUG:teuthology.orchestra.run.smithi187:> true 2023-10-12T18:40:37.609 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-12T18:40:37.609 INFO:teuthology.run_tasks:Running task clock... 2023-10-12T18:40:37.622 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-12T18:40:37.622 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-12T18:40:37.623 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T18:40:37.625 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-12T18:40:37.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T18:40:37.652 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-12T18:40:37.666 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-12T18:40:37.678 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-12T18:40:37.692 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-12T18:40:37.700 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2023-10-12T18:40:37.711 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2023-10-12T18:40:37.725 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-12T18:40:37.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2023-10-12T18:40:37.736 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2023-10-12T18:40:37.739 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-12T18:40:37.750 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-12T18:40:37.764 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-12T18:40:37.804 INFO:teuthology.orchestra.run.smithi170.stderr:bash: ntpq: command not found 2023-10-12T18:40:37.806 INFO:teuthology.orchestra.run.smithi170.stdout:210 Number of sources = 4 2023-10-12T18:40:37.807 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T18:40:37.807 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-10-12T18:40:37.807 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T18:40:37.807 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T18:40:37.807 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T18:40:37.808 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T18:40:37.832 INFO:teuthology.orchestra.run.smithi187.stderr:bash: ntpq: command not found 2023-10-12T18:40:37.835 INFO:teuthology.orchestra.run.smithi187.stdout:210 Number of sources = 4 2023-10-12T18:40:37.836 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T18:40:37.836 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2023-10-12T18:40:37.836 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T18:40:37.836 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T18:40:37.836 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T18:40:37.837 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T18:40:37.837 INFO:teuthology.run_tasks:Running task install... 2023-10-12T18:40:37.846 DEBUG:teuthology.task.install:project ceph 2023-10-12T18:40:37.847 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a'}} 2023-10-12T18:40:37.847 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a'} 2023-10-12T18:40:37.847 INFO:teuthology.task.install:Using flavor: default 2023-10-12T18:40:37.853 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-12T18:40:37.854 INFO:teuthology.task.install:extra packages: [] 2023-10-12T18:40:37.855 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a', 'tag': None, 'wait_for_package': False} 2023-10-12T18:40:37.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=84698e743199450ee5f6969d22ebe6bed3a68e4a 2023-10-12T18:40:37.857 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a', 'tag': None, 'wait_for_package': False} 2023-10-12T18:40:37.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=84698e743199450ee5f6969d22ebe6bed3a68e4a 2023-10-12T18:40:38.016 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-11-0853-pacific/84698e743199450ee5f6969d22ebe6bed3a68e4a/centos/8/flavors/default/ 2023-10-12T18:40:38.016 INFO:teuthology.task.install.rpm:Package version is 16.2.14-75.g84698e74 2023-10-12T18:40:38.197 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-11-0853-pacific/84698e743199450ee5f6969d22ebe6bed3a68e4a/centos/8/flavors/default/ 2023-10-12T18:40:38.198 INFO:teuthology.task.install.rpm:Package version is 16.2.14-75.g84698e74 2023-10-12T18:40:38.428 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-11-0853-pacific/84698e743199450ee5f6969d22ebe6bed3a68e4a/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-11-0853-pacific/84698e743199450ee5f6969d22ebe6bed3a68e4a/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-11-0853-pacific/84698e743199450ee5f6969d22ebe6bed3a68e4a/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-12T18:40:38.429 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:40:38.429 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-12T18:40:38.441 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-11-0853-pacific/84698e743199450ee5f6969d22ebe6bed3a68e4a/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-11-0853-pacific/84698e743199450ee5f6969d22ebe6bed3a68e4a/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-11-0853-pacific/84698e743199450ee5f6969d22ebe6bed3a68e4a/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-12T18:40:38.442 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:40:38.442 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-12T18:40:38.459 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-12T18:40:38.460 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/84698e743199450ee5f6969d22ebe6bed3a68e4a/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-12T18:40:38.469 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-12T18:40:38.470 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/84698e743199450ee5f6969d22ebe6bed3a68e4a/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-12T18:40:38.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-12T18:40:38.536 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-12T18:40:38.607 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-12T18:40:38.612 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-12T18:40:38.672 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2023-10-12T18:40:38.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2023-10-12T18:40:38.675 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2023-10-12T18:40:38.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2023-10-12T18:40:39.066 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:40:39.068 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:40:39.349 INFO:teuthology.orchestra.run.smithi170.stdout:44 files removed 2023-10-12T18:40:39.366 INFO:teuthology.orchestra.run.smithi187.stdout:44 files removed 2023-10-12T18:40:39.399 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw 2023-10-12T18:40:39.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw 2023-10-12T18:40:39.749 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:40:39.766 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:40:41.051 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 88 kB/s | 76 kB 00:00 2023-10-12T18:40:41.111 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 84 kB/s | 76 kB 00:00 2023-10-12T18:40:41.688 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 26 kB/s | 15 kB 00:00 2023-10-12T18:40:41.756 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 25 kB/s | 15 kB 00:00 2023-10-12T18:40:42.276 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 2.6 kB/s | 1.4 kB 00:00 2023-10-12T18:40:42.342 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 2.6 kB/s | 1.4 kB 00:00 2023-10-12T18:40:42.415 INFO:teuthology.orchestra.run.smithi170.stdout:Copr repo for python3-asyncssh owned by ceph 32 kB/s | 3.5 kB 00:00 2023-10-12T18:40:42.506 INFO:teuthology.orchestra.run.smithi187.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-10-12T18:40:43.800 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-10-12T18:40:47.085 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 1.2 MB/s | 24 kB 00:00 2023-10-12T18:40:47.092 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 16 MB 00:04 2023-10-12T18:40:47.481 INFO:teuthology.orchestra.run.smithi170.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 121 MB/s | 38 MB 00:00 2023-10-12T18:40:50.400 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-10-12T18:40:50.802 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 103 MB/s | 33 MB 00:00 2023-10-12T18:40:52.966 INFO:teuthology.orchestra.run.smithi170.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 108 MB/s | 33 MB 00:00 2023-10-12T18:40:55.940 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 120 MB/s | 38 MB 00:00 2023-10-12T18:41:01.269 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:41:01.272 INFO:teuthology.orchestra.run.smithi170.stdout:=================================================================================================== 2023-10-12T18:41:01.272 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-12T18:41:01.272 INFO:teuthology.orchestra.run.smithi170.stdout:=================================================================================================== 2023-10-12T18:41:01.273 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:41:01.273 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:16.2.14-75.g84698e74.el8 ceph 12 M 2023-10-12T18:41:01.273 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2023-10-12T18:41:01.273 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:16.2.14-75.g84698e74.el8 ceph 3.7 M 2023-10-12T18:41:01.273 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:16.2.14-75.g84698e74.el8 ceph 4.2 M 2023-10-12T18:41:01.274 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-12T18:41:01.274 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:16.2.14-75.g84698e74.el8 ceph 6.1 M 2023-10-12T18:41:01.274 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:16.2.14-75.g84698e74.el8 ceph 24 M 2023-10-12T18:41:01.274 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:16.2.14-75.g84698e74.el8 ceph 26 k 2023-10-12T18:41:01.274 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-12T18:41:01.275 INFO:teuthology.orchestra.run.smithi170.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-10-12T18:41:01.275 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:16.2.14-75.g84698e74.el8 ceph 824 k 2023-10-12T18:41:01.275 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-12T18:41:01.275 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-12T18:41:01.275 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:16.2.14-75.g84698e74.el8 ceph 493 k 2023-10-12T18:41:01.276 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-12T18:41:01.276 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:16.2.14-75.g84698e74.el8 ceph 3.8 M 2023-10-12T18:41:01.276 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-12T18:41:01.276 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-12T18:41:01.276 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:16.2.14-75.g84698e74.el8 ceph 45 k 2023-10-12T18:41:01.277 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:16.2.14-75.g84698e74.el8 ceph 99 k 2023-10-12T18:41:01.277 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:16.2.14-75.g84698e74.el8 ceph 215 k 2023-10-12T18:41:01.277 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-12T18:41:01.277 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:16.2.14-75.g84698e74.el8 ceph 390 k 2023-10-12T18:41:01.278 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:16.2.14-75.g84698e74.el8 ceph 369 k 2023-10-12T18:41:01.278 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:16.2.14-75.g84698e74.el8 ceph 114 k 2023-10-12T18:41:01.278 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:41:01.278 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:41:01.278 INFO:teuthology.orchestra.run.smithi170.stdout:=================================================================================================== 2023-10-12T18:41:01.279 INFO:teuthology.orchestra.run.smithi170.stdout:Install 21 Packages 2023-10-12T18:41:01.279 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 2 Packages 2023-10-12T18:41:01.279 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:41:01.280 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 57 M 2023-10-12T18:41:01.280 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:41:01.754 INFO:teuthology.orchestra.run.smithi170.stdout:(1/23): ceph-base-16.2.14-75.g84698e74.el8.x86_ 14 MB/s | 6.1 MB 00:00 2023-10-12T18:41:01.775 INFO:teuthology.orchestra.run.smithi170.stdout:(2/23): ceph-selinux-16.2.14-75.g84698e74.el8.x 1.2 MB/s | 26 kB 00:00 2023-10-12T18:41:01.820 INFO:teuthology.orchestra.run.smithi170.stdout:(3/23): libcephfs2-16.2.14-75.g84698e74.el8.x86 18 MB/s | 824 kB 00:00 2023-10-12T18:41:01.849 INFO:teuthology.orchestra.run.smithi170.stdout:(4/23): libradosstriper1-16.2.14-75.g84698e74.e 17 MB/s | 493 kB 00:00 2023-10-12T18:41:02.042 INFO:teuthology.orchestra.run.smithi170.stdout:(5/23): librgw2-16.2.14-75.g84698e74.el8.x86_64 20 MB/s | 3.8 MB 00:00 2023-10-12T18:41:02.064 INFO:teuthology.orchestra.run.smithi170.stdout:(6/23): python3-ceph-argparse-16.2.14-75.g84698 2.1 MB/s | 45 kB 00:00 2023-10-12T18:41:02.085 INFO:teuthology.orchestra.run.smithi170.stdout:(7/23): python3-ceph-common-16.2.14-75.g84698e7 4.5 MB/s | 99 kB 00:00 2023-10-12T18:41:02.108 INFO:teuthology.orchestra.run.smithi170.stdout:(8/23): python3-cephfs-16.2.14-75.g84698e74.el8 9.5 MB/s | 215 kB 00:00 2023-10-12T18:41:02.133 INFO:teuthology.orchestra.run.smithi170.stdout:(9/23): python3-rados-16.2.14-75.g84698e74.el8. 15 MB/s | 390 kB 00:00 2023-10-12T18:41:02.157 INFO:teuthology.orchestra.run.smithi170.stdout:(10/23): python3-rbd-16.2.14-75.g84698e74.el8.x 15 MB/s | 369 kB 00:00 2023-10-12T18:41:02.179 INFO:teuthology.orchestra.run.smithi170.stdout:(11/23): python3-rgw-16.2.14-75.g84698e74.el8.x 5.2 MB/s | 114 kB 00:00 2023-10-12T18:41:03.429 INFO:teuthology.orchestra.run.smithi170.stdout:(12/23): ceph-radosgw-16.2.14-75.g84698e74.el8. 5.5 MB/s | 12 MB 00:02 2023-10-12T18:41:03.869 INFO:teuthology.orchestra.run.smithi170.stdout:(13/23): gperftools-libs-2.7-9.el8.x86_64.rpm 181 kB/s | 306 kB 00:01 2023-10-12T18:41:04.119 INFO:teuthology.orchestra.run.smithi170.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 235 kB/s | 59 kB 00:00 2023-10-12T18:41:04.377 INFO:teuthology.orchestra.run.smithi170.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 291 kB/s | 75 kB 00:00 2023-10-12T18:41:04.417 INFO:teuthology.orchestra.run.smithi170.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.2 MB/s | 47 kB 00:00 2023-10-12T18:41:04.434 INFO:teuthology.orchestra.run.smithi170.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 2.5 MB/s | 44 kB 00:00 2023-10-12T18:41:04.454 INFO:teuthology.orchestra.run.smithi170.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2023-10-12T18:41:04.474 INFO:teuthology.orchestra.run.smithi170.stdout:(19/23): librdkafka-0.11.4-1.el8.x86_64.rpm 17 MB/s | 353 kB 00:00 2023-10-12T18:41:04.747 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:41:04.749 INFO:teuthology.orchestra.run.smithi187.stdout:=================================================================================================== 2023-10-12T18:41:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-10-12T18:41:04.750 INFO:teuthology.orchestra.run.smithi187.stdout:=================================================================================================== 2023-10-12T18:41:04.750 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:41:04.750 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:16.2.14-75.g84698e74.el8 ceph 12 M 2023-10-12T18:41:04.751 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2023-10-12T18:41:04.751 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:16.2.14-75.g84698e74.el8 ceph 3.7 M 2023-10-12T18:41:04.751 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:16.2.14-75.g84698e74.el8 ceph 4.2 M 2023-10-12T18:41:04.751 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-10-12T18:41:04.751 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:16.2.14-75.g84698e74.el8 ceph 6.1 M 2023-10-12T18:41:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:16.2.14-75.g84698e74.el8 ceph 24 M 2023-10-12T18:41:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:16.2.14-75.g84698e74.el8 ceph 26 k 2023-10-12T18:41:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-12T18:41:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-10-12T18:41:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:16.2.14-75.g84698e74.el8 ceph 824 k 2023-10-12T18:41:04.753 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-12T18:41:04.753 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-12T18:41:04.753 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:16.2.14-75.g84698e74.el8 ceph 493 k 2023-10-12T18:41:04.753 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-12T18:41:04.753 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:16.2.14-75.g84698e74.el8 ceph 3.8 M 2023-10-12T18:41:04.754 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-12T18:41:04.754 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-12T18:41:04.754 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:16.2.14-75.g84698e74.el8 ceph 45 k 2023-10-12T18:41:04.754 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:16.2.14-75.g84698e74.el8 ceph 99 k 2023-10-12T18:41:04.754 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:16.2.14-75.g84698e74.el8 ceph 215 k 2023-10-12T18:41:04.755 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-12T18:41:04.755 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:16.2.14-75.g84698e74.el8 ceph 390 k 2023-10-12T18:41:04.755 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:16.2.14-75.g84698e74.el8 ceph 369 k 2023-10-12T18:41:04.755 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:16.2.14-75.g84698e74.el8 ceph 114 k 2023-10-12T18:41:04.755 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:41:04.756 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:41:04.756 INFO:teuthology.orchestra.run.smithi187.stdout:=================================================================================================== 2023-10-12T18:41:04.756 INFO:teuthology.orchestra.run.smithi187.stdout:Install 21 Packages 2023-10-12T18:41:04.756 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 2 Packages 2023-10-12T18:41:04.757 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:41:04.757 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 57 M 2023-10-12T18:41:04.758 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:41:04.782 INFO:teuthology.orchestra.run.smithi170.stdout:(20/23): librados2-16.2.14-75.g84698e74.el8.x86 12 MB/s | 3.7 MB 00:00 2023-10-12T18:41:05.043 INFO:teuthology.orchestra.run.smithi170.stdout:(21/23): librbd1-16.2.14-75.g84698e74.el8.x86_6 16 MB/s | 4.2 MB 00:00 2023-10-12T18:41:05.097 INFO:teuthology.orchestra.run.smithi170.stdout:(22/23): leveldb-1.22-1.el8.x86_64.rpm 108 kB/s | 181 kB 00:01 2023-10-12T18:41:05.269 INFO:teuthology.orchestra.run.smithi170.stdout:(23/23): ceph-common-16.2.14-75.g84698e74.el8.x 6.1 MB/s | 24 MB 00:03 2023-10-12T18:41:05.273 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:41:05.274 INFO:teuthology.orchestra.run.smithi170.stdout:Total 14 MB/s | 57 MB 00:03 2023-10-12T18:41:05.277 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:41:05.322 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:41:05.323 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:41:05.645 INFO:teuthology.orchestra.run.smithi187.stdout:(1/23): ceph-base-16.2.14-75.g84698e74.el8.x86_ 9.7 MB/s | 6.1 MB 00:00 2023-10-12T18:41:05.676 INFO:teuthology.orchestra.run.smithi187.stdout:(2/23): ceph-selinux-16.2.14-75.g84698e74.el8.x 825 kB/s | 26 kB 00:00 2023-10-12T18:41:05.765 INFO:teuthology.orchestra.run.smithi187.stdout:(3/23): libcephfs2-16.2.14-75.g84698e74.el8.x86 9.1 MB/s | 824 kB 00:00 2023-10-12T18:41:05.815 INFO:teuthology.orchestra.run.smithi187.stdout:(4/23): libradosstriper1-16.2.14-75.g84698e74.e 9.8 MB/s | 493 kB 00:00 2023-10-12T18:41:05.914 INFO:teuthology.orchestra.run.smithi187.stdout:(5/23): ceph-radosgw-16.2.14-75.g84698e74.el8.x 13 MB/s | 12 MB 00:00 2023-10-12T18:41:05.952 INFO:teuthology.orchestra.run.smithi187.stdout:(6/23): python3-ceph-argparse-16.2.14-75.g84698 1.2 MB/s | 45 kB 00:00 2023-10-12T18:41:05.986 INFO:teuthology.orchestra.run.smithi187.stdout:(7/23): python3-ceph-common-16.2.14-75.g84698e7 2.8 MB/s | 99 kB 00:00 2023-10-12T18:41:06.027 INFO:teuthology.orchestra.run.smithi187.stdout:(8/23): librgw2-16.2.14-75.g84698e74.el8.x86_64 18 MB/s | 3.8 MB 00:00 2023-10-12T18:41:06.028 INFO:teuthology.orchestra.run.smithi187.stdout:(9/23): python3-cephfs-16.2.14-75.g84698e74.el8 5.0 MB/s | 215 kB 00:00 2023-10-12T18:41:06.052 INFO:teuthology.orchestra.run.smithi187.stdout:(10/23): python3-rados-16.2.14-75.g84698e74.el8 15 MB/s | 390 kB 00:00 2023-10-12T18:41:06.064 INFO:teuthology.orchestra.run.smithi187.stdout:(11/23): python3-rbd-16.2.14-75.g84698e74.el8.x 10 MB/s | 369 kB 00:00 2023-10-12T18:41:06.074 INFO:teuthology.orchestra.run.smithi187.stdout:(12/23): python3-rgw-16.2.14-75.g84698e74.el8.x 5.2 MB/s | 114 kB 00:00 2023-10-12T18:41:06.095 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:41:06.127 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:41:06.312 INFO:teuthology.orchestra.run.smithi187.stdout:(13/23): leveldb-1.22-1.el8.x86_64.rpm 762 kB/s | 181 kB 00:00 2023-10-12T18:41:06.362 INFO:teuthology.orchestra.run.smithi187.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2023-10-12T18:41:06.382 INFO:teuthology.orchestra.run.smithi187.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 965 kB/s | 306 kB 00:00 2023-10-12T18:41:06.418 INFO:teuthology.orchestra.run.smithi187.stdout:(16/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-10-12T18:41:06.427 INFO:teuthology.orchestra.run.smithi187.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 982 kB/s | 44 kB 00:00 2023-10-12T18:41:06.449 INFO:teuthology.orchestra.run.smithi187.stdout:(18/23): librdkafka-0.11.4-1.el8.x86_64.rpm 16 MB/s | 353 kB 00:00 2023-10-12T18:41:06.466 INFO:teuthology.orchestra.run.smithi187.stdout:(19/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.3 MB/s | 259 kB 00:00 2023-10-12T18:41:06.473 INFO:teuthology.orchestra.run.smithi187.stdout:(20/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.0 MB/s | 47 kB 00:00 2023-10-12T18:41:06.759 INFO:teuthology.orchestra.run.smithi187.stdout:(21/23): librados2-16.2.14-75.g84698e74.el8.x86 13 MB/s | 3.7 MB 00:00 2023-10-12T18:41:06.777 INFO:teuthology.orchestra.run.smithi187.stdout:(22/23): librbd1-16.2.14-75.g84698e74.el8.x86_6 14 MB/s | 4.2 MB 00:00 2023-10-12T18:41:07.232 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:41:07.396 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-10-12T18:41:07.783 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-10-12T18:41:08.004 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:16.2.14-75.g84698e74.el8.x86_64 2/25 2023-10-12T18:41:08.108 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.14-75.g84698e74.el8.x86_64 2/25 2023-10-12T18:41:08.294 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:16.2.14-75.g84698e74.el8.x86_64 3/25 2023-10-12T18:41:08.544 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-10-12T18:41:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 5/25 2023-10-12T18:41:09.145 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 5/25 2023-10-12T18:41:09.329 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:16.2.14-75.g84698e74.el8.x86_64 6/25 2023-10-12T18:41:09.412 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.14-75.g84698e74.el8.x86_64 6/25 2023-10-12T18:41:09.579 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/25 2023-10-12T18:41:09.647 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/25 2023-10-12T18:41:10.113 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-10-12T18:41:10.270 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:16.2.14-75.g84698e74.el8.x86_64 9/25 2023-10-12T18:41:10.349 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:16.2.14-75.g84698e74.el8.x86_64 9/25 2023-10-12T18:41:10.510 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-10-12T18:41:10.635 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:16.2.14-75.g84698e74.el8.x 11/25 2023-10-12T18:41:10.815 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:16.2.14-75.g84698e74.el8.x86_64 12/25 2023-10-12T18:41:11.022 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:16.2.14-75.g84698e74.el8.x86_64 13/25 2023-10-12T18:41:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:16.2.14-75.g84698e74.el8.x86_64 14/25 2023-10-12T18:41:11.412 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:16.2.14-75.g84698e74.el8.x86_64 15/25 2023-10-12T18:41:11.468 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:16.2.14-75.g84698e74.el8.x86_64 15/25 2023-10-12T18:41:11.681 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-12T18:41:11.846 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-10-12T18:41:12.076 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-12T18:41:12.237 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:16.2.14-75.g84698e74.el8.x86 19/25 2023-10-12T18:41:13.916 INFO:teuthology.orchestra.run.smithi187.stdout:(23/23): ceph-common-16.2.14-75.g84698e74.el8.x 2.7 MB/s | 24 MB 00:08 2023-10-12T18:41:13.920 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:41:13.920 INFO:teuthology.orchestra.run.smithi187.stdout:Total 6.3 MB/s | 57 MB 00:09 2023-10-12T18:41:13.925 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:41:13.973 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:41:13.973 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:41:14.738 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:41:14.742 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:41:15.818 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:41:16.033 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-10-12T18:41:16.420 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-10-12T18:41:16.608 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:16.2.14-75.g84698e74.el8.x86_64 2/25 2023-10-12T18:41:16.719 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.2.14-75.g84698e74.el8.x86_64 2/25 2023-10-12T18:41:16.935 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:16.2.14-75.g84698e74.el8.x86_64 3/25 2023-10-12T18:41:17.088 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 20/25 2023-10-12T18:41:17.231 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-10-12T18:41:17.303 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 20/25 2023-10-12T18:41:17.483 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 5/25 2023-10-12T18:41:17.896 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 5/25 2023-10-12T18:41:18.014 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 20/25 2023-10-12T18:41:18.082 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:16.2.14-75.g84698e74.el8.x86_64 6/25 2023-10-12T18:41:18.150 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.2.14-75.g84698e74.el8.x86_64 6/25 2023-10-12T18:41:18.320 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 21/25 2023-10-12T18:41:18.371 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-10-12T18:41:18.407 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 21/25 2023-10-12T18:41:18.519 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:16.2.14-75.g84698e74.el8.x86_64 22/25 2023-10-12T18:41:18.624 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-10-12T18:41:18.987 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-10-12T18:41:19.140 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:16.2.14-75.g84698e74.el8.x86_64 9/25 2023-10-12T18:41:19.225 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:16.2.14-75.g84698e74.el8.x86_64 9/25 2023-10-12T18:41:19.414 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-10-12T18:41:19.554 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:16.2.14-75.g84698e74.el8.x 11/25 2023-10-12T18:41:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:16.2.14-75.g84698e74.el8.x86_64 12/25 2023-10-12T18:41:20.040 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:16.2.14-75.g84698e74.el8.x86_64 13/25 2023-10-12T18:41:20.307 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:16.2.14-75.g84698e74.el8.x86_64 14/25 2023-10-12T18:41:20.499 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:16.2.14-75.g84698e74.el8.x86_64 15/25 2023-10-12T18:41:20.553 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:16.2.14-75.g84698e74.el8.x86_64 15/25 2023-10-12T18:41:20.734 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-12T18:41:20.942 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-10-12T18:41:21.198 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-12T18:41:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:16.2.14-75.g84698e74.el8.x86 19/25 2023-10-12T18:41:26.576 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 20/25 2023-10-12T18:41:26.823 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 20/25 2023-10-12T18:41:27.540 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 20/25 2023-10-12T18:41:27.874 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 21/25 2023-10-12T18:41:27.967 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 21/25 2023-10-12T18:41:28.106 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:16.2.14-75.g84698e74.el8.x86_64 22/25 2023-10-12T18:41:36.379 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:16.2.14-75.g84698e74.el8.x86_64 22/25 2023-10-12T18:41:36.380 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2023-10-12T18:41:36.380 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2023-10-12T18:41:36.380 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /dev 2023-10-12T18:41:36.380 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /run 2023-10-12T18:41:36.380 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2023-10-12T18:41:36.381 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2023-10-12T18:41:36.381 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2023-10-12T18:41:36.381 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2023-10-12T18:41:36.381 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /dev 2023-10-12T18:41:36.381 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:41:37.567 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:16.2.14-75.g84698e74.el8.x86_64 23/25 2023-10-12T18:41:37.643 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-75.g84698e74.el8.x86_64 23/25 2023-10-12T18:41:37.644 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-12T18:41:37.843 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-12T18:41:37.844 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T18:41:38.607 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T18:41:38.608 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 1/25 2023-10-12T18:41:38.608 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 2/25 2023-10-12T18:41:38.609 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:16.2.14-75.g84698e74.el8.x86_64 3/25 2023-10-12T18:41:38.609 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:16.2.14-75.g84698e74.el8.x86_64 4/25 2023-10-12T18:41:38.609 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 5/25 2023-10-12T18:41:38.610 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:16.2.14-75.g84698e74.el8.x86_64 6/25 2023-10-12T18:41:38.610 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:16.2.14-75.g84698e74.el8.x86_64 7/25 2023-10-12T18:41:38.610 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:16.2.14-75.g84698e74.el8.x 8/25 2023-10-12T18:41:38.610 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:16.2.14-75.g84698e74.el8.x86 9/25 2023-10-12T18:41:38.610 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:16.2.14-75.g84698e74.el8.x86_64 10/25 2023-10-12T18:41:38.611 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:16.2.14-75.g84698e74.el8.x86_64 11/25 2023-10-12T18:41:38.611 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:16.2.14-75.g84698e74.el8.x86_64 12/25 2023-10-12T18:41:38.611 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:16.2.14-75.g84698e74.el8.x86_64 13/25 2023-10-12T18:41:38.611 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-10-12T18:41:38.611 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2023-10-12T18:41:38.611 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2023-10-12T18:41:38.612 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2023-10-12T18:41:38.612 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-10-12T18:41:38.612 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-10-12T18:41:38.612 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2023-10-12T18:41:38.612 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/25 2023-10-12T18:41:38.613 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.14-75.g84698e74.el8.x86_64 22/25 2023-10-12T18:41:38.613 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-12T18:41:38.613 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.14-75.g84698e74.el8.x86_64 24/25 2023-10-12T18:41:39.970 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T18:41:39.971 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:47 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:41:41.868 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:41:41.973 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:41:41.973 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2023-10-12T18:41:41.974 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.974 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.974 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:41:41.974 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.974 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.975 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.975 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.975 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-12T18:41:41.975 INFO:teuthology.orchestra.run.smithi170.stdout: leveldb-1.22-1.el8.x86_64 2023-10-12T18:41:41.975 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.975 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-12T18:41:41.976 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-12T18:41:41.976 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.976 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-12T18:41:41.976 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.976 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-12T18:41:41.977 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-12T18:41:41.977 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.977 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.977 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.977 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-12T18:41:41.978 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.978 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.978 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:41.978 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:41:41.979 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:41:42.125 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test 2023-10-12T18:41:42.494 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:41:43.267 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:51 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:41:45.063 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:41:45.064 INFO:teuthology.orchestra.run.smithi170.stdout:=========================================================================================== 2023-10-12T18:41:45.064 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-12T18:41:45.064 INFO:teuthology.orchestra.run.smithi170.stdout:=========================================================================================== 2023-10-12T18:41:45.064 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:41:45.064 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:16.2.14-75.g84698e74.el8 ceph 52 M 2023-10-12T18:41:45.065 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-12T18:41:45.065 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-12T18:41:45.065 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:16.2.14-75.g84698e74.el8 ceph 174 k 2023-10-12T18:41:45.065 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-12T18:41:45.065 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-12T18:41:45.066 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-12T18:41:45.066 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:41:45.066 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:41:45.066 INFO:teuthology.orchestra.run.smithi170.stdout:=========================================================================================== 2023-10-12T18:41:45.067 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-10-12T18:41:45.067 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:41:45.067 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 53 M 2023-10-12T18:41:45.068 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 196 M 2023-10-12T18:41:45.068 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:41:45.069 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 19 MB/s | 70 kB 00:00 2023-10-12T18:41:45.102 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.8 MB/s | 161 kB 00:00 2023-10-12T18:41:45.120 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 10 MB/s | 187 kB 00:00 2023-10-12T18:41:45.138 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 17 MB/s | 302 kB 00:00 2023-10-12T18:41:45.470 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): libcephsqlite-16.2.14-75.g84698e74.el8.x 430 kB/s | 174 kB 00:00 2023-10-12T18:41:45.614 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:16.2.14-75.g84698e74.el8.x86_64 22/25 2023-10-12T18:41:45.614 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2023-10-12T18:41:45.614 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2023-10-12T18:41:45.615 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2023-10-12T18:41:45.615 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /run 2023-10-12T18:41:45.615 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2023-10-12T18:41:45.615 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2023-10-12T18:41:45.615 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2023-10-12T18:41:45.616 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2023-10-12T18:41:45.616 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2023-10-12T18:41:45.616 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:41:46.820 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:16.2.14-75.g84698e74.el8.x86_64 23/25 2023-10-12T18:41:46.892 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-75.g84698e74.el8.x86_64 23/25 2023-10-12T18:41:46.892 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-12T18:41:47.121 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-12T18:41:47.121 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T18:41:47.948 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T18:41:47.948 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 1/25 2023-10-12T18:41:47.949 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 2/25 2023-10-12T18:41:47.949 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:16.2.14-75.g84698e74.el8.x86_64 3/25 2023-10-12T18:41:47.949 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:16.2.14-75.g84698e74.el8.x86_64 4/25 2023-10-12T18:41:47.949 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 5/25 2023-10-12T18:41:47.950 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:16.2.14-75.g84698e74.el8.x86_64 6/25 2023-10-12T18:41:47.950 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:16.2.14-75.g84698e74.el8.x86_64 7/25 2023-10-12T18:41:47.950 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:16.2.14-75.g84698e74.el8.x 8/25 2023-10-12T18:41:47.950 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:16.2.14-75.g84698e74.el8.x86 9/25 2023-10-12T18:41:47.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:16.2.14-75.g84698e74.el8.x86_64 10/25 2023-10-12T18:41:47.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:16.2.14-75.g84698e74.el8.x86_64 11/25 2023-10-12T18:41:47.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:16.2.14-75.g84698e74.el8.x86_64 12/25 2023-10-12T18:41:47.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:16.2.14-75.g84698e74.el8.x86_64 13/25 2023-10-12T18:41:47.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-10-12T18:41:47.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2023-10-12T18:41:47.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2023-10-12T18:41:47.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2023-10-12T18:41:47.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-12T18:41:47.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-10-12T18:41:47.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-10-12T18:41:47.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-10-12T18:41:47.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.2.14-75.g84698e74.el8.x86_64 22/25 2023-10-12T18:41:47.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-12T18:41:47.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.2.14-75.g84698e74.el8.x86_64 24/25 2023-10-12T18:41:48.817 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): ceph-test-16.2.14-75.g84698e74.el8.x86_6 14 MB/s | 52 MB 00:03 2023-10-12T18:41:48.819 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:41:48.820 INFO:teuthology.orchestra.run.smithi170.stdout:Total 14 MB/s | 53 MB 00:03 2023-10-12T18:41:48.823 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:41:48.845 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:41:48.845 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:41:49.190 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:41:49.194 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:41:49.391 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T18:41:49.392 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:54 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:41:49.692 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:41:49.892 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-10-12T18:41:50.098 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-10-12T18:41:50.176 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-10-12T18:41:50.332 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-10-12T18:41:50.532 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-10-12T18:41:50.664 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:16.2.14-75.g84698e74.el8.x86_64 5/6 2023-10-12T18:41:51.398 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:41:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:41:51.510 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2023-10-12T18:41:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.511 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:41:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.512 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-12T18:41:51.512 INFO:teuthology.orchestra.run.smithi187.stdout: leveldb-1.22-1.el8.x86_64 2023-10-12T18:41:51.512 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.512 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-12T18:41:51.512 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-12T18:41:51.513 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.513 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-12T18:41:51.513 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.513 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-12T18:41:51.513 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-12T18:41:51.513 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.514 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.514 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.514 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-12T18:41:51.514 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.514 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.515 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:51.515 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:41:51.515 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:41:51.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-test 2023-10-12T18:41:52.022 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:41:52.938 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:57 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:41:54.831 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:41:54.831 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================== 2023-10-12T18:41:54.832 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-10-12T18:41:54.832 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================== 2023-10-12T18:41:54.832 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:41:54.832 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:16.2.14-75.g84698e74.el8 ceph 52 M 2023-10-12T18:41:54.832 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-10-12T18:41:54.833 INFO:teuthology.orchestra.run.smithi187.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-12T18:41:54.833 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:16.2.14-75.g84698e74.el8 ceph 174 k 2023-10-12T18:41:54.833 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-12T18:41:54.833 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-12T18:41:54.833 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-12T18:41:54.834 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:41:54.834 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:41:54.834 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================== 2023-10-12T18:41:54.834 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2023-10-12T18:41:54.834 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:41:54.835 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 53 M 2023-10-12T18:41:54.835 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 196 M 2023-10-12T18:41:54.836 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:41:54.837 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 18 MB/s | 70 kB 00:00 2023-10-12T18:41:54.870 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.9 MB/s | 161 kB 00:00 2023-10-12T18:41:54.887 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 11 MB/s | 187 kB 00:00 2023-10-12T18:41:54.905 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 17 MB/s | 302 kB 00:00 2023-10-12T18:41:54.989 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): libcephsqlite-16.2.14-75.g84698e74.el8.x 1.1 MB/s | 174 kB 00:00 2023-10-12T18:41:55.320 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:16.2.14-75.g84698e74.el8.x86_64 5/6 2023-10-12T18:41:55.578 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:16.2.14-75.g84698e74.el8.x86_64 6/6 2023-10-12T18:41:55.844 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-2:16.2.14-75.g84698e74.el8.x86_64 6/6 2023-10-12T18:41:55.844 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:16.2.14-75.g84698e74.el8.x86_64 1/6 2023-10-12T18:41:55.844 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:16.2.14-75.g84698e74.el8.x86_64 2/6 2023-10-12T18:41:55.844 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-10-12T18:41:55.845 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-10-12T18:41:55.845 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-10-12T18:41:56.900 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-10-12T18:41:56.901 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:04 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:41:57.352 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): ceph-test-16.2.14-75.g84698e74.el8.x86_6 21 MB/s | 52 MB 00:02 2023-10-12T18:41:57.355 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:41:57.355 INFO:teuthology.orchestra.run.smithi187.stdout:Total 21 MB/s | 53 MB 00:02 2023-10-12T18:41:57.359 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:41:57.382 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:41:57.382 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:41:57.733 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:41:57.737 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:41:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:41:58.470 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-10-12T18:41:58.693 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-10-12T18:41:58.756 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:41:58.779 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-10-12T18:41:58.855 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:41:58.856 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:41:58.856 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:58.857 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.5-12.el8.x86_64 2023-10-12T18:41:58.857 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:41:58.857 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-12T18:41:58.857 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-12T18:41:58.857 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-12T18:41:58.858 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:41:58.858 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:41:58.935 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-10-12T18:41:58.973 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph 2023-10-12T18:41:59.115 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-10-12T18:41:59.267 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:16.2.14-75.g84698e74.el8.x86_64 5/6 2023-10-12T18:41:59.328 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:00.216 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:08 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:02.084 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:42:02.086 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================================== 2023-10-12T18:42:02.087 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-12T18:42:02.087 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================================== 2023-10-12T18:42:02.087 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:42:02.088 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:16.2.14-75.g84698e74.el8 ceph 6.4 k 2023-10-12T18:42:02.088 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-12T18:42:02.088 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:16.2.14-75.g84698e74.el8 ceph 2.4 M 2023-10-12T18:42:02.088 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:16.2.14-75.g84698e74.el8 ceph 1.6 M 2023-10-12T18:42:02.089 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 244 k 2023-10-12T18:42:02.089 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:16.2.14-75.g84698e74.el8 ceph 5.0 M 2023-10-12T18:42:02.089 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:16.2.14-75.g84698e74.el8 ceph 18 M 2023-10-12T18:42:02.089 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-12T18:42:02.089 INFO:teuthology.orchestra.run.smithi170.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-12T18:42:02.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-12T18:42:02.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-12T18:42:02.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-12T18:42:02.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-12T18:42:02.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-12T18:42:02.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-12T18:42:02.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-12T18:42:02.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-12T18:42:02.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-12T18:42:02.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-12T18:42:02.092 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-12T18:42:02.092 INFO:teuthology.orchestra.run.smithi170.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-12T18:42:02.092 INFO:teuthology.orchestra.run.smithi170.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-12T18:42:02.092 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-12T18:42:02.092 INFO:teuthology.orchestra.run.smithi170.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-12T18:42:02.092 INFO:teuthology.orchestra.run.smithi170.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-12T18:42:02.093 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-12T18:42:02.093 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-12T18:42:02.093 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-12T18:42:02.093 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-12T18:42:02.093 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:02.094 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:42:02.094 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================================== 2023-10-12T18:42:02.094 INFO:teuthology.orchestra.run.smithi170.stdout:Install 28 Packages 2023-10-12T18:42:02.094 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:02.095 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 30 M 2023-10-12T18:42:02.095 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 96 M 2023-10-12T18:42:02.095 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:42:02.542 INFO:teuthology.orchestra.run.smithi170.stdout:(1/28): ceph-16.2.14-75.g84698e74.el8.x86_64.rp 22 kB/s | 6.4 kB 00:00 2023-10-12T18:42:03.050 INFO:teuthology.orchestra.run.smithi170.stdout:(2/28): ceph-mgr-16.2.14-75.g84698e74.el8.x86_6 2.0 MB/s | 1.6 MB 00:00 2023-10-12T18:42:03.112 INFO:teuthology.orchestra.run.smithi170.stdout:(3/28): ceph-mds-16.2.14-75.g84698e74.el8.x86_6 2.8 MB/s | 2.4 MB 00:00 2023-10-12T18:42:03.217 INFO:teuthology.orchestra.run.smithi170.stdout:(4/28): ceph-mgr-modules-core-16.2.14-75.g84698 2.3 MB/s | 244 kB 00:00 2023-10-12T18:42:03.376 INFO:teuthology.orchestra.run.smithi170.stdout:(5/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 276 kB/s | 44 kB 00:00 2023-10-12T18:42:03.413 INFO:teuthology.orchestra.run.smithi170.stdout:(6/28): ceph-mon-16.2.14-75.g84698e74.el8.x86_6 5.8 MB/s | 5.0 MB 00:00 2023-10-12T18:42:03.496 INFO:teuthology.orchestra.run.smithi170.stdout:(7/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.5 MB/s | 185 kB 00:00 2023-10-12T18:42:03.577 INFO:teuthology.orchestra.run.smithi170.stdout:(8/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 4.6 MB/s | 384 kB 00:00 2023-10-12T18:42:03.613 INFO:teuthology.orchestra.run.smithi170.stdout:(9/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 861 kB/s | 173 kB 00:00 2023-10-12T18:42:03.617 INFO:teuthology.orchestra.run.smithi170.stdout:(10/28): python3-jaraco-6.2-6.el8.noarch.rpm 278 kB/s | 11 kB 00:00 2023-10-12T18:42:03.653 INFO:teuthology.orchestra.run.smithi170.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 439 kB/s | 18 kB 00:00 2023-10-12T18:42:03.658 INFO:teuthology.orchestra.run.smithi170.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 1.2 MB/s | 49 kB 00:00 2023-10-12T18:42:03.694 INFO:teuthology.orchestra.run.smithi170.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 1.4 MB/s | 59 kB 00:00 2023-10-12T18:42:03.701 INFO:teuthology.orchestra.run.smithi170.stdout:(14/28): python3-pecan-1.3.2-9.el8.noarch.rpm 6.5 MB/s | 283 kB 00:00 2023-10-12T18:42:03.734 INFO:teuthology.orchestra.run.smithi170.stdout:(15/28): python3-portend-2.6-1.el8.noarch.rpm 392 kB/s | 16 kB 00:00 2023-10-12T18:42:03.741 INFO:teuthology.orchestra.run.smithi170.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 471 kB/s | 19 kB 00:00 2023-10-12T18:42:03.774 INFO:teuthology.orchestra.run.smithi170.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 599 kB/s | 24 kB 00:00 2023-10-12T18:42:03.782 INFO:teuthology.orchestra.run.smithi170.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 715 kB/s | 29 kB 00:00 2023-10-12T18:42:03.814 INFO:teuthology.orchestra.run.smithi170.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 691 kB/s | 27 kB 00:00 2023-10-12T18:42:03.825 INFO:teuthology.orchestra.run.smithi170.stdout:(20/28): python3-waitress-1.4.3-1.el8.noarch.rp 5.8 MB/s | 255 kB 00:00 2023-10-12T18:42:03.866 INFO:teuthology.orchestra.run.smithi170.stdout:(21/28): python3-webtest-2.0.33-1.el8.noarch.rp 2.0 MB/s | 85 kB 00:00 2023-10-12T18:42:03.897 INFO:teuthology.orchestra.run.smithi170.stdout:(22/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2023-10-12T18:42:03.901 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:16.2.14-75.g84698e74.el8.x86_64 5/6 2023-10-12T18:42:03.907 INFO:teuthology.orchestra.run.smithi170.stdout:(23/28): python3-zc-lockfile-2.0-2.el8.noarch.r 560 kB/s | 23 kB 00:00 2023-10-12T18:42:03.939 INFO:teuthology.orchestra.run.smithi170.stdout:(24/28): python3-werkzeug-0.12.2-4.el8.noarch.r 11 MB/s | 457 kB 00:00 2023-10-12T18:42:03.945 INFO:teuthology.orchestra.run.smithi170.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 1.0 MB/s | 39 kB 00:00 2023-10-12T18:42:03.958 INFO:teuthology.orchestra.run.smithi170.stdout:(26/28): python3-mako-1.0.6-13.el8.noarch.rpm 7.9 MB/s | 157 kB 00:00 2023-10-12T18:42:03.963 INFO:teuthology.orchestra.run.smithi170.stdout:(27/28): python3-pytz-2017.2-9.el8.noarch.rpm 3.0 MB/s | 54 kB 00:00 2023-10-12T18:42:04.148 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:16.2.14-75.g84698e74.el8.x86_64 6/6 2023-10-12T18:42:04.327 INFO:teuthology.orchestra.run.smithi170.stdout:(28/28): ceph-osd-16.2.14-75.g84698e74.el8.x86_ 14 MB/s | 18 MB 00:01 2023-10-12T18:42:04.330 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:42:04.330 INFO:teuthology.orchestra.run.smithi170.stdout:Total 13 MB/s | 30 MB 00:02 2023-10-12T18:42:04.334 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:42:04.372 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:42:04.372 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:42:04.418 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-test-2:16.2.14-75.g84698e74.el8.x86_64 6/6 2023-10-12T18:42:04.418 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:16.2.14-75.g84698e74.el8.x86_64 1/6 2023-10-12T18:42:04.419 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:16.2.14-75.g84698e74.el8.x86_64 2/6 2023-10-12T18:42:04.419 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-10-12T18:42:04.419 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-10-12T18:42:04.419 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-10-12T18:42:04.751 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:42:04.756 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:42:05.575 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-10-12T18:42:05.575 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:10 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:05.795 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:42:05.993 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-10-12T18:42:06.201 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-10-12T18:42:06.389 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-10-12T18:42:06.710 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-10-12T18:42:07.051 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-10-12T18:42:07.262 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-10-12T18:42:07.467 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-10-12T18:42:07.637 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-10-12T18:42:07.641 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:42:07.740 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:07.740 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:42:07.740 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:07.741 INFO:teuthology.orchestra.run.smithi187.stdout: jq-1.5-12.el8.x86_64 2023-10-12T18:42:07.741 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:07.741 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-12T18:42:07.741 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-12T18:42:07.741 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-12T18:42:07.742 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:07.742 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:42:07.818 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-10-12T18:42:07.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph 2023-10-12T18:42:08.008 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-10-12T18:42:08.164 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-10-12T18:42:08.204 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:08.331 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-10-12T18:42:08.467 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-10-12T18:42:08.698 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-10-12T18:42:08.898 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-10-12T18:42:09.100 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-10-12T18:42:09.231 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:14 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:09.339 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-10-12T18:42:09.568 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-10-12T18:42:09.780 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-10-12T18:42:10.080 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-10-12T18:42:10.340 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-10-12T18:42:10.622 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-10-12T18:42:10.942 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-75.g84698e74.el8.n 23/28 2023-10-12T18:42:11.065 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:42:11.068 INFO:teuthology.orchestra.run.smithi187.stdout:====================================================================================================== 2023-10-12T18:42:11.068 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-10-12T18:42:11.068 INFO:teuthology.orchestra.run.smithi187.stdout:====================================================================================================== 2023-10-12T18:42:11.068 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:42:11.069 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:16.2.14-75.g84698e74.el8 ceph 6.4 k 2023-10-12T18:42:11.069 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-10-12T18:42:11.069 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:16.2.14-75.g84698e74.el8 ceph 2.4 M 2023-10-12T18:42:11.069 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:16.2.14-75.g84698e74.el8 ceph 1.6 M 2023-10-12T18:42:11.069 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 244 k 2023-10-12T18:42:11.070 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:16.2.14-75.g84698e74.el8 ceph 5.0 M 2023-10-12T18:42:11.070 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:16.2.14-75.g84698e74.el8 ceph 18 M 2023-10-12T18:42:11.070 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-12T18:42:11.070 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-12T18:42:11.070 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-12T18:42:11.071 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-12T18:42:11.071 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-12T18:42:11.071 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-12T18:42:11.071 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-12T18:42:11.071 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-12T18:42:11.072 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-12T18:42:11.072 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-12T18:42:11.072 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-12T18:42:11.072 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-12T18:42:11.072 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-12T18:42:11.072 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-12T18:42:11.073 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-12T18:42:11.073 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-12T18:42:11.073 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-12T18:42:11.073 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-12T18:42:11.073 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-12T18:42:11.074 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-12T18:42:11.074 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-12T18:42:11.074 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-12T18:42:11.074 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:11.075 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:42:11.075 INFO:teuthology.orchestra.run.smithi187.stdout:====================================================================================================== 2023-10-12T18:42:11.075 INFO:teuthology.orchestra.run.smithi187.stdout:Install 28 Packages 2023-10-12T18:42:11.075 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:11.076 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 30 M 2023-10-12T18:42:11.076 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 96 M 2023-10-12T18:42:11.076 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:42:11.112 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 24/28 2023-10-12T18:42:11.404 INFO:teuthology.orchestra.run.smithi187.stdout:(1/28): ceph-16.2.14-75.g84698e74.el8.x86_64.rp 35 kB/s | 6.4 kB 00:00 2023-10-12T18:42:11.768 INFO:teuthology.orchestra.run.smithi187.stdout:(2/28): ceph-mds-16.2.14-75.g84698e74.el8.x86_6 4.5 MB/s | 2.4 MB 00:00 2023-10-12T18:42:12.032 INFO:teuthology.orchestra.run.smithi187.stdout:(3/28): ceph-mon-16.2.14-75.g84698e74.el8.x86_6 8.0 MB/s | 5.0 MB 00:00 2023-10-12T18:42:12.094 INFO:teuthology.orchestra.run.smithi187.stdout:(4/28): ceph-mgr-modules-core-16.2.14-75.g84698 3.8 MB/s | 244 kB 00:00 2023-10-12T18:42:12.319 INFO:teuthology.orchestra.run.smithi187.stdout:(5/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 194 kB/s | 44 kB 00:00 2023-10-12T18:42:12.497 INFO:teuthology.orchestra.run.smithi187.stdout:(6/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.0 MB/s | 185 kB 00:00 2023-10-12T18:42:12.530 INFO:teuthology.orchestra.run.smithi187.stdout:(7/28): ceph-mgr-16.2.14-75.g84698e74.el8.x86_6 1.2 MB/s | 1.6 MB 00:01 2023-10-12T18:42:12.586 INFO:teuthology.orchestra.run.smithi187.stdout:(8/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.9 MB/s | 173 kB 00:00 2023-10-12T18:42:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 24/28 2023-10-12T18:42:12.646 INFO:teuthology.orchestra.run.smithi187.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 186 kB/s | 11 kB 00:00 2023-10-12T18:42:12.769 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:16.2.14-75.g84698e74.el8.x86_64 25/28 2023-10-12T18:42:12.820 INFO:teuthology.orchestra.run.smithi187.stdout:(10/28): ceph-osd-16.2.14-75.g84698e74.el8.x86_ 17 MB/s | 18 MB 00:01 2023-10-12T18:42:12.821 INFO:teuthology.orchestra.run.smithi187.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 101 kB/s | 18 kB 00:00 2023-10-12T18:42:12.885 INFO:teuthology.orchestra.run.smithi187.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 752 kB/s | 49 kB 00:00 2023-10-12T18:42:12.990 INFO:teuthology.orchestra.run.smithi187.stdout:(13/28): python3-cherrypy-18.4.0-1.el8.noarch.r 836 kB/s | 384 kB 00:00 2023-10-12T18:42:13.038 INFO:teuthology.orchestra.run.smithi187.stdout:(14/28): python3-pecan-1.3.2-9.el8.noarch.rpm 2.6 MB/s | 283 kB 00:00 2023-10-12T18:42:13.046 INFO:teuthology.orchestra.run.smithi187.stdout:(15/28): python3-portend-2.6-1.el8.noarch.rpm 275 kB/s | 16 kB 00:00 2023-10-12T18:42:13.049 INFO:teuthology.orchestra.run.smithi187.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 326 kB/s | 19 kB 00:00 2023-10-12T18:42:13.062 INFO:teuthology.orchestra.run.smithi187.stdout:(17/28): python3-more-itertools-7.2.0-3.el8.noa 243 kB/s | 59 kB 00:00 2023-10-12T18:42:13.104 INFO:teuthology.orchestra.run.smithi187.stdout:(18/28): python3-singledispatch-3.4.0.3-18.el8. 416 kB/s | 24 kB 00:00 2023-10-12T18:42:13.108 INFO:teuthology.orchestra.run.smithi187.stdout:(19/28): python3-tempora-1.14.1-5.el8.noarch.rp 493 kB/s | 29 kB 00:00 2023-10-12T18:42:13.136 INFO:teuthology.orchestra.run.smithi187.stdout:(20/28): python3-trustme-0.6.0-4.el8.noarch.rpm 373 kB/s | 27 kB 00:00 2023-10-12T18:42:13.188 INFO:teuthology.orchestra.run.smithi187.stdout:(21/28): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2023-10-12T18:42:13.197 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:16.2.14-75.g84698e74.el8.x86_64 25/28 2023-10-12T18:42:13.197 INFO:teuthology.orchestra.run.smithi187.stdout:(22/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.8 MB/s | 251 kB 00:00 2023-10-12T18:42:13.246 INFO:teuthology.orchestra.run.smithi187.stdout:(23/28): python3-werkzeug-0.12.2-4.el8.noarch.r 9.3 MB/s | 457 kB 00:00 2023-10-12T18:42:13.246 INFO:teuthology.orchestra.run.smithi187.stdout:(24/28): python3-zc-lockfile-2.0-2.el8.noarch.r 397 kB/s | 23 kB 00:00 2023-10-12T18:42:13.249 INFO:teuthology.orchestra.run.smithi187.stdout:(25/28): python3-webtest-2.0.33-1.el8.noarch.rp 760 kB/s | 85 kB 00:00 2023-10-12T18:42:13.277 INFO:teuthology.orchestra.run.smithi187.stdout:(26/28): python3-markupsafe-0.23-19.el8.x86_64. 1.2 MB/s | 39 kB 00:00 2023-10-12T18:42:13.298 INFO:teuthology.orchestra.run.smithi187.stdout:(27/28): python3-mako-1.0.6-13.el8.noarch.rpm 3.0 MB/s | 157 kB 00:00 2023-10-12T18:42:13.303 INFO:teuthology.orchestra.run.smithi187.stdout:(28/28): python3-pytz-2017.2-9.el8.noarch.rpm 1.0 MB/s | 54 kB 00:00 2023-10-12T18:42:13.305 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:42:13.305 INFO:teuthology.orchestra.run.smithi187.stdout:Total 14 MB/s | 30 MB 00:02 2023-10-12T18:42:13.309 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:42:13.346 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:42:13.346 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:42:13.353 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:16.2.14-75.g84698e74.el8.x86_64 26/28 2023-10-12T18:42:13.572 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:16.2.14-75.g84698e74.el8.x86_64 26/28 2023-10-12T18:42:13.728 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:16.2.14-75.g84698e74.el8.x86_64 27/28 2023-10-12T18:42:13.736 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:42:13.741 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:42:13.746 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:16.2.14-75.g84698e74.el8.x86_64 27/28 2023-10-12T18:42:13.843 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:16.2.14-75.g84698e74.el8.x86_64 28/28 2023-10-12T18:42:14.245 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-2:16.2.14-75.g84698e74.el8.x86_64 28/28 2023-10-12T18:42:14.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:16.2.14-75.g84698e74.el8.x86_64 1/28 2023-10-12T18:42:14.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:16.2.14-75.g84698e74.el8.x86_64 2/28 2023-10-12T18:42:14.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 3/28 2023-10-12T18:42:14.247 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:16.2.14-75.g84698e74.el8.x86_64 4/28 2023-10-12T18:42:14.247 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:16.2.14-75.g84698e74.el8.x86_64 5/28 2023-10-12T18:42:14.247 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-75.g84698e74.el8.n 6/28 2023-10-12T18:42:14.247 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-10-12T18:42:14.247 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-10-12T18:42:14.248 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-10-12T18:42:14.248 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-10-12T18:42:14.248 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-10-12T18:42:14.248 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-10-12T18:42:14.248 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-10-12T18:42:14.248 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-10-12T18:42:14.249 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-10-12T18:42:14.249 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-10-12T18:42:14.249 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-10-12T18:42:14.249 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-10-12T18:42:14.249 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-10-12T18:42:14.250 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-10-12T18:42:14.250 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-10-12T18:42:14.250 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-10-12T18:42:14.250 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-10-12T18:42:14.250 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-10-12T18:42:14.251 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-10-12T18:42:14.251 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-10-12T18:42:14.251 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-10-12T18:42:14.815 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:42:15.084 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-10-12T18:42:15.355 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-10-12T18:42:15.519 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-10-12T18:42:15.720 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-10-12T18:42:15.720 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:23 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:15.805 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-10-12T18:42:16.147 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-10-12T18:42:16.348 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-10-12T18:42:16.579 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-10-12T18:42:16.747 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-10-12T18:42:16.946 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-10-12T18:42:17.118 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-10-12T18:42:17.291 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-10-12T18:42:17.475 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-10-12T18:42:17.653 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-10-12T18:42:17.706 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:42:17.807 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:17.807 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:42:17.807 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:17.808 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:17.808 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:17.808 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:42:17.808 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:17.808 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:17.809 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-12T18:42:17.809 INFO:teuthology.orchestra.run.smithi170.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-12T18:42:17.809 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-12T18:42:17.809 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-12T18:42:17.809 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-12T18:42:17.809 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-12T18:42:17.810 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-12T18:42:17.810 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-12T18:42:17.810 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-12T18:42:17.810 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-12T18:42:17.811 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-12T18:42:17.811 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-2.6-1.el8.noarch 2023-10-12T18:42:17.811 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-12T18:42:17.811 INFO:teuthology.orchestra.run.smithi170.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-12T18:42:17.811 INFO:teuthology.orchestra.run.smithi170.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-12T18:42:17.812 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-12T18:42:17.812 INFO:teuthology.orchestra.run.smithi170.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-12T18:42:17.812 INFO:teuthology.orchestra.run.smithi170.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-12T18:42:17.812 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-12T18:42:17.812 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-12T18:42:17.813 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-12T18:42:17.813 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-12T18:42:17.813 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:17.813 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:42:17.861 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-10-12T18:42:17.953 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base 2023-10-12T18:42:18.109 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-10-12T18:42:18.286 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-10-12T18:42:18.318 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:18.543 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-10-12T18:42:18.786 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-10-12T18:42:18.999 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-10-12T18:42:19.153 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:27 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:19.319 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-10-12T18:42:19.622 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-10-12T18:42:19.892 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-10-12T18:42:20.242 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-75.g84698e74.el8.n 23/28 2023-10-12T18:42:20.461 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:42:20.848 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 24/28 2023-10-12T18:42:20.931 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:42:20.932 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:42:20.932 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:42:21.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install cephadm 2023-10-12T18:42:21.401 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:22.300 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:30 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:22.345 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 24/28 2023-10-12T18:42:22.513 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:16.2.14-75.g84698e74.el8.x86_64 25/28 2023-10-12T18:42:22.942 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:16.2.14-75.g84698e74.el8.x86_64 25/28 2023-10-12T18:42:23.113 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:16.2.14-75.g84698e74.el8.x86_64 26/28 2023-10-12T18:42:23.332 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:16.2.14-75.g84698e74.el8.x86_64 26/28 2023-10-12T18:42:23.488 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:16.2.14-75.g84698e74.el8.x86_64 27/28 2023-10-12T18:42:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:16.2.14-75.g84698e74.el8.x86_64 27/28 2023-10-12T18:42:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:16.2.14-75.g84698e74.el8.x86_64 28/28 2023-10-12T18:42:24.002 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-2:16.2.14-75.g84698e74.el8.x86_64 28/28 2023-10-12T18:42:24.002 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:16.2.14-75.g84698e74.el8.x86_64 1/28 2023-10-12T18:42:24.002 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:16.2.14-75.g84698e74.el8.x86_64 2/28 2023-10-12T18:42:24.003 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 3/28 2023-10-12T18:42:24.003 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:16.2.14-75.g84698e74.el8.x86_64 4/28 2023-10-12T18:42:24.003 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:16.2.14-75.g84698e74.el8.x86_64 5/28 2023-10-12T18:42:24.003 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-75.g84698e74.el8.n 6/28 2023-10-12T18:42:24.003 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-10-12T18:42:24.003 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-10-12T18:42:24.004 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-10-12T18:42:24.004 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-10-12T18:42:24.004 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-10-12T18:42:24.004 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-10-12T18:42:24.005 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-10-12T18:42:24.005 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-10-12T18:42:24.005 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-10-12T18:42:24.005 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-10-12T18:42:24.006 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-10-12T18:42:24.006 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-10-12T18:42:24.006 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-10-12T18:42:24.006 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-10-12T18:42:24.007 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-10-12T18:42:24.007 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-10-12T18:42:24.007 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-10-12T18:42:24.007 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-10-12T18:42:24.007 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-10-12T18:42:24.007 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-10-12T18:42:24.008 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-10-12T18:42:24.098 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:42:24.099 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:42:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-12T18:42:24.100 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:42:24.100 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:42:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 84 k 2023-10-12T18:42:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:24.101 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:42:24.101 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:42:24.101 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-12T18:42:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:24.101 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 84 k 2023-10-12T18:42:24.102 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 348 k 2023-10-12T18:42:24.102 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:42:24.246 INFO:teuthology.orchestra.run.smithi170.stdout:cephadm-16.2.14-75.g84698e74.el8.noarch.rpm 578 kB/s | 84 kB 00:00 2023-10-12T18:42:24.247 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:42:24.247 INFO:teuthology.orchestra.run.smithi170.stdout:Total 572 kB/s | 84 kB 00:00 2023-10-12T18:42:24.248 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:42:24.255 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:42:24.255 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:42:24.256 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:42:24.261 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:42:24.320 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:42:25.677 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-10-12T18:42:25.677 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:30 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:27.123 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/1 2023-10-12T18:42:27.289 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/1 2023-10-12T18:42:27.495 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/1 2023-10-12T18:42:27.641 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:42:27.742 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:27.743 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:42:27.743 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:27.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:27.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:27.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:42:27.745 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:27.745 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:27.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-12T18:42:27.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-12T18:42:27.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-12T18:42:27.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-12T18:42:27.747 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-12T18:42:27.747 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-12T18:42:27.747 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-12T18:42:27.747 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-12T18:42:27.748 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-12T18:42:27.748 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-12T18:42:27.748 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-12T18:42:27.748 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-2.6-1.el8.noarch 2023-10-12T18:42:27.749 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-12T18:42:27.749 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-12T18:42:27.749 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-12T18:42:27.750 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-12T18:42:27.750 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-12T18:42:27.750 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-12T18:42:27.750 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-12T18:42:27.751 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-12T18:42:27.751 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-12T18:42:27.751 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-12T18:42:27.751 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:27.751 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:42:27.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-base 2023-10-12T18:42:28.260 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:28.414 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/1 2023-10-12T18:42:28.414 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:36 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:30.105 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:35 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:31.771 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:42:31.867 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:31.867 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:42:31.867 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:42:31.868 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:31.868 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:42:32.005 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache 2023-10-12T18:42:32.380 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:32.650 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-base-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:42:33.148 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:42:33.149 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:42:33.149 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:42:33.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install cephadm 2023-10-12T18:42:33.619 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:33.927 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:41 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:35.360 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:40 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:35.770 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:42:35.770 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:42:35.771 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-12T18:42:35.771 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:42:35.771 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:42:35.771 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-75.g84698e74.el8 ceph 161 k 2023-10-12T18:42:35.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:35.772 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:42:35.772 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:42:35.772 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-12T18:42:35.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:35.773 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 161 k 2023-10-12T18:42:35.773 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 470 k 2023-10-12T18:42:35.773 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:42:36.154 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-16.2.14-75.g84698e7 421 kB/s | 161 kB 00:00 2023-10-12T18:42:36.155 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:42:36.155 INFO:teuthology.orchestra.run.smithi170.stdout:Total 419 kB/s | 161 kB 00:00 2023-10-12T18:42:36.156 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:42:36.168 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:42:36.168 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:42:36.221 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:42:36.226 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:42:36.344 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:42:36.507 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-75.g84698e74.e 1/1 2023-10-12T18:42:36.914 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-75.g84698e74.e 1/1 2023-10-12T18:42:37.177 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:42:37.178 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:42:37.178 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-10-12T18:42:37.179 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:42:37.179 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:42:37.179 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 84 k 2023-10-12T18:42:37.179 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:37.180 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:42:37.180 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:42:37.180 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-10-12T18:42:37.180 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:37.180 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 84 k 2023-10-12T18:42:37.181 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 348 k 2023-10-12T18:42:37.181 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:42:37.544 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-16.2.14-75.g84698e74.el8.noarch.rpm 231 kB/s | 84 kB 00:00 2023-10-12T18:42:37.544 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:42:37.544 INFO:teuthology.orchestra.run.smithi187.stdout:Total 230 kB/s | 84 kB 00:00 2023-10-12T18:42:37.545 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:42:37.552 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:42:37.552 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:42:37.554 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:42:37.558 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:42:37.603 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:42:37.900 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-75.g84698e74.e 1/1 2023-10-12T18:42:37.900 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:45 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:40.061 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:42:40.158 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:40.158 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:42:40.158 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:40.159 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:40.159 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:42:40.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr 2023-10-12T18:42:40.643 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:41.232 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/1 2023-10-12T18:42:41.374 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/1 2023-10-12T18:42:41.495 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:49 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:41.579 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/1 2023-10-12T18:42:42.622 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/1 2023-10-12T18:42:42.622 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:47 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:42.791 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:42:43.272 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:42:43.274 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:42:43.274 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:42:43.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-dashboard 2023-10-12T18:42:43.745 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:44.599 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:52 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:44.710 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:42:44.806 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:44.807 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:42:44.807 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:42:44.807 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:44.807 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:42:44.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-immutable-object-cache 2023-10-12T18:42:45.323 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:46.273 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:51 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:46.383 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:42:46.383 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================================== 2023-10-12T18:42:46.384 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-12T18:42:46.384 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================================== 2023-10-12T18:42:46.384 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:42:46.384 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 3.1 M 2023-10-12T18:42:46.384 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-12T18:42:46.385 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 24 k 2023-10-12T18:42:46.385 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 15 k 2023-10-12T18:42:46.385 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-12T18:42:46.385 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-12T18:42:46.385 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-12T18:42:46.386 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:46.386 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:42:46.386 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================================== 2023-10-12T18:42:46.386 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-10-12T18:42:46.386 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:46.387 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.4 M 2023-10-12T18:42:46.387 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 22 M 2023-10-12T18:42:46.387 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:42:46.566 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): ceph-grafana-dashboards-16.2.14-75.g8469 219 kB/s | 24 kB 00:00 2023-10-12T18:42:46.646 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): ceph-prometheus-alerts-16.2.14-75.g84698 78 kB/s | 15 kB 00:00 2023-10-12T18:42:46.815 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 135 kB/s | 34 kB 00:00 2023-10-12T18:42:46.847 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-10-12T18:42:46.903 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): ceph-mgr-dashboard-16.2.14-75.g84698e74. 7.0 MB/s | 3.1 MB 00:00 2023-10-12T18:42:46.911 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 741 kB/s | 196 kB 00:00 2023-10-12T18:42:46.913 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:42:46.914 INFO:teuthology.orchestra.run.smithi170.stdout:Total 6.5 MB/s | 3.4 MB 00:00 2023-10-12T18:42:46.915 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:42:46.924 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:42:46.924 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:42:47.122 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:42:47.126 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:42:47.410 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:42:47.591 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-10-12T18:42:47.764 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-10-12T18:42:47.933 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-10-12T18:42:48.053 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-75.g84698e74.el8.no 4/6 2023-10-12T18:42:48.116 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:42:48.117 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:42:48.118 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2023-10-12T18:42:48.118 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:42:48.118 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:42:48.118 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-75.g84698e74.el8 ceph 161 k 2023-10-12T18:42:48.118 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:48.119 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:42:48.119 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:42:48.119 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-10-12T18:42:48.119 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:48.119 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 161 k 2023-10-12T18:42:48.120 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 470 k 2023-10-12T18:42:48.120 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:42:48.379 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-immutable-object-cache-16.2.14-75.g84698e7 619 kB/s | 161 kB 00:00 2023-10-12T18:42:48.379 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:42:48.380 INFO:teuthology.orchestra.run.smithi187.stdout:Total 615 kB/s | 161 kB 00:00 2023-10-12T18:42:48.380 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:42:48.392 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:42:48.393 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:42:48.446 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:42:48.450 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:42:48.582 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:42:48.733 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-75.g84698e74.e 1/1 2023-10-12T18:42:49.149 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-75.g84698e74.e 1/1 2023-10-12T18:42:49.181 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-75.g84698e74.el8.n 5/6 2023-10-12T18:42:49.431 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-75.g84698e74.el8.noarch 6/6 2023-10-12T18:42:49.715 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-75.g84698e74.el8.noarch 6/6 2023-10-12T18:42:49.715 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-75.g84698e74.el8.n 1/6 2023-10-12T18:42:49.715 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-75.g84698e74.el8.noarch 2/6 2023-10-12T18:42:49.716 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-75.g84698e74.el8.no 3/6 2023-10-12T18:42:49.716 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-10-12T18:42:49.716 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-10-12T18:42:50.378 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-75.g84698e74.e 1/1 2023-10-12T18:42:50.378 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:55 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:50.657 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-10-12T18:42:50.657 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:58 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:53.747 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:42:53.843 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:53.844 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:42:53.844 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:42:53.844 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:42:53.844 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:42:53.845 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-12T18:42:53.845 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-12T18:42:53.845 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-12T18:42:53.845 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:53.845 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:42:53.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-12T18:42:54.124 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:42:54.222 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:54.223 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:42:54.223 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:42:54.223 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:42:54.223 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:42:54.332 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:54.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr 2023-10-12T18:42:54.744 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:55.302 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:03 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:42:55.761 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:00 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:57.077 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:42:57.078 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================================================== 2023-10-12T18:42:57.079 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-12T18:42:57.079 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================================================== 2023-10-12T18:42:57.079 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:42:57.079 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 8.6 M 2023-10-12T18:42:57.079 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-12T18:42:57.080 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-12T18:42:57.080 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-12T18:42:57.080 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-12T18:42:57.080 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-12T18:42:57.080 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-12T18:42:57.081 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-12T18:42:57.081 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-12T18:42:57.081 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:57.081 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:42:57.081 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================================================== 2023-10-12T18:42:57.082 INFO:teuthology.orchestra.run.smithi170.stdout:Install 8 Packages 2023-10-12T18:42:57.082 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:42:57.082 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 37 M 2023-10-12T18:42:57.083 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 199 M 2023-10-12T18:42:57.083 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:42:57.126 INFO:teuthology.orchestra.run.smithi170.stdout:(1/8): libquadmath-8.4.1-1.el8.x86_64.rpm 3.7 MB/s | 169 kB 00:00 2023-10-12T18:42:57.138 INFO:teuthology.orchestra.run.smithi170.stdout:(2/8): libgfortran-8.4.1-1.el8.x86_64.rpm 11 MB/s | 639 kB 00:00 2023-10-12T18:42:57.162 INFO:teuthology.orchestra.run.smithi170.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 9.1 MB/s | 225 kB 00:00 2023-10-12T18:42:57.187 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-mgr-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:42:57.231 INFO:teuthology.orchestra.run.smithi170.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 54 MB/s | 3.7 MB 00:00 2023-10-12T18:42:57.398 INFO:teuthology.orchestra.run.smithi170.stdout:(5/8): python3-scipy-1.0.0-20.module+el8.1.0+33 53 MB/s | 15 MB 00:00 2023-10-12T18:42:57.431 INFO:teuthology.orchestra.run.smithi170.stdout:(6/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2023-10-12T18:42:57.474 INFO:teuthology.orchestra.run.smithi170.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 61 MB/s | 4.6 MB 00:00 2023-10-12T18:42:57.660 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:42:57.661 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:42:57.661 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:42:57.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-dashboard 2023-10-12T18:42:58.073 INFO:teuthology.orchestra.run.smithi170.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-75 8.7 MB/s | 8.6 MB 00:00 2023-10-12T18:42:58.074 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:42:58.075 INFO:teuthology.orchestra.run.smithi170.stdout:Total 37 MB/s | 37 MB 00:00 2023-10-12T18:42:58.084 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:42:58.104 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:42:58.104 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:42:58.126 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:42:58.440 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:42:58.444 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:42:59.061 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:42:59.211 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-10-12T18:42:59.334 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-10-12T18:42:59.355 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:04 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:42:59.486 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-10-12T18:43:00.021 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-10-12T18:43:00.186 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-10-12T18:43:00.725 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-10-12T18:43:00.961 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-10-12T18:43:01.188 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:43:01.190 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================================== 2023-10-12T18:43:01.190 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-10-12T18:43:01.190 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================================== 2023-10-12T18:43:01.190 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:43:01.190 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 3.1 M 2023-10-12T18:43:01.191 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-10-12T18:43:01.191 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 24 k 2023-10-12T18:43:01.191 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 15 k 2023-10-12T18:43:01.191 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-12T18:43:01.191 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-12T18:43:01.192 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-12T18:43:01.192 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:01.192 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:43:01.192 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================================== 2023-10-12T18:43:01.192 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2023-10-12T18:43:01.193 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:01.193 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 3.4 M 2023-10-12T18:43:01.193 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 22 M 2023-10-12T18:43:01.194 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:43:01.418 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): ceph-grafana-dashboards-16.2.14-75.g8469 125 kB/s | 24 kB 00:00 2023-10-12T18:43:01.419 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): ceph-prometheus-alerts-16.2.14-75.g84698 76 kB/s | 15 kB 00:00 2023-10-12T18:43:01.521 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): ceph-mgr-dashboard-16.2.14-75.g84698e74. 11 MB/s | 3.1 MB 00:00 2023-10-12T18:43:01.524 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 317 kB/s | 34 kB 00:00 2023-10-12T18:43:01.564 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 993 kB/s | 43 kB 00:00 2023-10-12T18:43:01.580 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2023-10-12T18:43:01.583 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:43:01.584 INFO:teuthology.orchestra.run.smithi187.stdout:Total 8.7 MB/s | 3.4 MB 00:00 2023-10-12T18:43:01.585 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:43:01.594 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:43:01.594 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:43:01.720 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-10-12T18:43:01.787 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:43:01.791 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:43:02.047 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-10-12T18:43:02.105 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:43:02.253 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-10-12T18:43:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-10-12T18:43:02.523 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-10-12T18:43:02.733 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-10-12T18:43:02.862 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-75.g84698e74.el8.no 4/6 2023-10-12T18:43:04.022 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-75.g84698e74.el8.n 5/6 2023-10-12T18:43:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-75.g84698e74.el8.noarch 6/6 2023-10-12T18:43:04.345 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-10-12T18:43:04.559 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-75.g84698e74.el8.noarch 6/6 2023-10-12T18:43:04.559 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-75.g84698e74.el8.n 1/6 2023-10-12T18:43:04.560 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-75.g84698e74.el8.noarch 2/6 2023-10-12T18:43:04.560 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-75.g84698e74.el8.no 3/6 2023-10-12T18:43:04.560 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-10-12T18:43:04.560 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-10-12T18:43:05.600 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-10-12T18:43:05.600 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:10 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:43:05.689 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-10-12T18:43:05.877 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-75.g84698e74 8/8 2023-10-12T18:43:06.171 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-75.g84698e74 8/8 2023-10-12T18:43:06.172 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-75.g84698e74 1/8 2023-10-12T18:43:06.172 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 2/8 2023-10-12T18:43:06.172 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 3/8 2023-10-12T18:43:06.172 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2023-10-12T18:43:06.173 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2023-10-12T18:43:06.173 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2023-10-12T18:43:06.173 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 7/8 2023-10-12T18:43:07.120 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 8/8 2023-10-12T18:43:07.121 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:15 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:07.711 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:43:07.807 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:07.807 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:43:07.808 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:43:07.808 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:43:07.808 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:43:07.808 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-12T18:43:07.808 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-12T18:43:07.808 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-12T18:43:07.809 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:07.809 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:43:07.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-12T18:43:08.302 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:09.213 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:43:09.250 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:14 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:43:09.310 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:09.310 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:43:09.310 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:43:09.310 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-12T18:43:09.311 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-12T18:43:09.311 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-12T18:43:09.311 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-12T18:43:09.311 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-12T18:43:09.311 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-12T18:43:09.312 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-12T18:43:09.312 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:09.312 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:43:09.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-rook 2023-10-12T18:43:09.829 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:10.679 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:18 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:11.369 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:43:11.370 INFO:teuthology.orchestra.run.smithi187.stdout:====================================================================================================================== 2023-10-12T18:43:11.370 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-10-12T18:43:11.371 INFO:teuthology.orchestra.run.smithi187.stdout:====================================================================================================================== 2023-10-12T18:43:11.371 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:43:11.371 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 8.6 M 2023-10-12T18:43:11.371 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-10-12T18:43:11.371 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-12T18:43:11.372 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-12T18:43:11.372 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-12T18:43:11.372 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-12T18:43:11.372 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-12T18:43:11.372 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-12T18:43:11.373 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-12T18:43:11.373 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:11.373 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:43:11.373 INFO:teuthology.orchestra.run.smithi187.stdout:====================================================================================================================== 2023-10-12T18:43:11.373 INFO:teuthology.orchestra.run.smithi187.stdout:Install 8 Packages 2023-10-12T18:43:11.374 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:11.374 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 37 M 2023-10-12T18:43:11.375 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 199 M 2023-10-12T18:43:11.375 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:43:11.409 INFO:teuthology.orchestra.run.smithi187.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 6.1 MB/s | 225 kB 00:00 2023-10-12T18:43:11.479 INFO:teuthology.orchestra.run.smithi187.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 54 MB/s | 3.7 MB 00:00 2023-10-12T18:43:11.558 INFO:teuthology.orchestra.run.smithi187.stdout:(3/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 60 MB/s | 4.7 MB 00:00 2023-10-12T18:43:11.699 INFO:teuthology.orchestra.run.smithi187.stdout:(4/8): python3-scipy-1.0.0-20.module+el8.1.0+33 45 MB/s | 15 MB 00:00 2023-10-12T18:43:11.718 INFO:teuthology.orchestra.run.smithi187.stdout:(5/8): libquadmath-8.4.1-1.el8.x86_64.rpm 8.6 MB/s | 169 kB 00:00 2023-10-12T18:43:11.744 INFO:teuthology.orchestra.run.smithi187.stdout:(6/8): openblas-0.3.12-1.el8.x86_64.rpm 25 MB/s | 4.6 MB 00:00 2023-10-12T18:43:11.750 INFO:teuthology.orchestra.run.smithi187.stdout:(7/8): libgfortran-8.4.1-1.el8.x86_64.rpm 20 MB/s | 639 kB 00:00 2023-10-12T18:43:12.269 INFO:teuthology.orchestra.run.smithi187.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-75 9.6 MB/s | 8.6 MB 00:00 2023-10-12T18:43:12.270 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:43:12.271 INFO:teuthology.orchestra.run.smithi187.stdout:Total 41 MB/s | 37 MB 00:00 2023-10-12T18:43:12.280 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:43:12.299 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:43:12.300 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:43:12.461 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:43:12.462 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================================= 2023-10-12T18:43:12.463 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-12T18:43:12.463 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================================= 2023-10-12T18:43:12.463 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:43:12.463 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 27 k 2023-10-12T18:43:12.463 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-12T18:43:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-12T18:43:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-12T18:43:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-12T18:43:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-12T18:43:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-12T18:43:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-12T18:43:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-12T18:43:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-12T18:43:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-12T18:43:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-12T18:43:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-12T18:43:12.466 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-12T18:43:12.466 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:12.466 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:43:12.466 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================================= 2023-10-12T18:43:12.466 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2023-10-12T18:43:12.467 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:12.467 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 1.9 M 2023-10-12T18:43:12.467 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 24 M 2023-10-12T18:43:12.468 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:43:12.662 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:43:12.666 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:43:13.139 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): ceph-mgr-rook-16.2.14-75.g84698e74.el8. 245 kB/s | 27 kB 00:00 2023-10-12T18:43:13.150 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 269 kB/s | 33 kB 00:00 2023-10-12T18:43:13.152 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): python3-certifi-2018.10.15-7.el8.noarch 130 kB/s | 16 kB 00:00 2023-10-12T18:43:13.253 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 639 kB/s | 64 kB 00:00 2023-10-12T18:43:13.314 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:43:13.333 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 770 kB/s | 61 kB 00:00 2023-10-12T18:43:13.374 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.7 MB/s | 155 kB 00:00 2023-10-12T18:43:13.382 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): python3-google-auth-1.1.1-10.el8.noarch 334 kB/s | 81 kB 00:00 2023-10-12T18:43:13.403 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.5 MB/s | 43 kB 00:00 2023-10-12T18:43:13.421 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 688 kB/s | 27 kB 00:00 2023-10-12T18:43:13.424 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 5.8 MB/s | 126 kB 00:00 2023-10-12T18:43:13.445 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): python3-jsonpointer-1.10-11.el8.noarch 955 kB/s | 20 kB 00:00 2023-10-12T18:43:13.448 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): python3-pyasn1-modules-0.3.7-6.el8.noa 4.1 MB/s | 110 kB 00:00 2023-10-12T18:43:13.470 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-10-12T18:43:13.604 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-10-12T18:43:13.795 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-10-12T18:43:13.979 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.4 MB/s | 1.2 MB 00:00 2023-10-12T18:43:13.980 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:43:13.980 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-10-12T18:43:13.987 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:43:14.003 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:43:14.003 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:43:14.093 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:43:14.097 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:43:14.346 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-10-12T18:43:14.528 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-10-12T18:43:14.584 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:43:14.813 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-12T18:43:15.048 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-10-12T18:43:15.070 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-10-12T18:43:15.241 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-10-12T18:43:15.278 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-10-12T18:43:15.400 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-10-12T18:43:15.633 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-10-12T18:43:15.890 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-12T18:43:16.021 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-10-12T18:43:16.154 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-12T18:43:16.375 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-10-12T18:43:16.576 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-10-12T18:43:16.769 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-10-12T18:43:16.908 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-12T18:43:17.126 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-10-12T18:43:18.272 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-10-12T18:43:18.542 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-10-12T18:43:18.691 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:16.2.14-75.g84698e74.el8.noarch 13/13 2023-10-12T18:43:18.840 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-10-12T18:43:18.961 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-75.g84698e74.el8.noarch 13/13 2023-10-12T18:43:18.961 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:16.2.14-75.g84698e74.el8.noarch 1/13 2023-10-12T18:43:18.961 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-10-12T18:43:18.962 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-10-12T18:43:18.962 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-10-12T18:43:18.962 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-10-12T18:43:18.962 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-10-12T18:43:18.962 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-10-12T18:43:18.963 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-12T18:43:18.963 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-12T18:43:18.963 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2023-10-12T18:43:18.963 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2023-10-12T18:43:18.963 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/13 2023-10-12T18:43:19.978 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-10-12T18:43:19.978 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:27 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:20.159 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-10-12T18:43:20.377 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-75.g84698e74 8/8 2023-10-12T18:43:20.660 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-75.g84698e74 8/8 2023-10-12T18:43:20.660 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-75.g84698e74 1/8 2023-10-12T18:43:20.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-10-12T18:43:20.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-10-12T18:43:20.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-10-12T18:43:20.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-10-12T18:43:20.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-10-12T18:43:20.662 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-10-12T18:43:21.781 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-10-12T18:43:21.781 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:26 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:43:22.079 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:43:22.179 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:22.180 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:43:22.180 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:43:22.180 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-12T18:43:22.180 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-12T18:43:22.181 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-12T18:43:22.181 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-12T18:43:22.181 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-12T18:43:22.181 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-12T18:43:22.181 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-12T18:43:22.182 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-12T18:43:22.182 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-12T18:43:22.182 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-12T18:43:22.182 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-12T18:43:22.182 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-12T18:43:22.183 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:22.183 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:43:22.491 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-cephadm 2023-10-12T18:43:22.862 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:23.733 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:31 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:23.952 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:43:24.052 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:24.053 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:43:24.053 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:43:24.053 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-12T18:43:24.053 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-12T18:43:24.053 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-12T18:43:24.054 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-12T18:43:24.054 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-12T18:43:24.054 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-12T18:43:24.054 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-12T18:43:24.054 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:24.055 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:43:24.192 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-rook 2023-10-12T18:43:24.558 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:25.584 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:43:25.584 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================== 2023-10-12T18:43:25.585 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-12T18:43:25.585 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================== 2023-10-12T18:43:25.585 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:43:25.585 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 116 k 2023-10-12T18:43:25.585 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-12T18:43:25.586 INFO:teuthology.orchestra.run.smithi170.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-10-12T18:43:25.586 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-12T18:43:25.586 INFO:teuthology.orchestra.run.smithi170.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-10-12T18:43:25.586 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-12T18:43:25.587 INFO:teuthology.orchestra.run.smithi170.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-10-12T18:43:25.587 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:25.587 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:43:25.587 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================== 2023-10-12T18:43:25.587 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-10-12T18:43:25.588 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:25.588 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 5.8 M 2023-10-12T18:43:25.588 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 24 M 2023-10-12T18:43:25.589 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:43:25.930 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 104 kB/s | 21 kB 00:00 2023-10-12T18:43:25.944 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): ceph-mgr-cephadm-16.2.14-75.g84698e74.el 535 kB/s | 116 kB 00:00 2023-10-12T18:43:26.031 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): python3-remoto-1.2.1-1.el8.noarch.rpm 487 kB/s | 49 kB 00:00 2023-10-12T18:43:26.069 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-babel-2.5.1-5.el8.noarch.rpm 38 MB/s | 4.8 MB 00:00 2023-10-12T18:43:26.109 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 928 kB/s | 355 kB 00:00 2023-10-12T18:43:26.129 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 5.4 MB/s | 538 kB 00:00 2023-10-12T18:43:26.131 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:43:26.132 INFO:teuthology.orchestra.run.smithi170.stdout:Total 11 MB/s | 5.8 MB 00:00 2023-10-12T18:43:26.135 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:43:26.144 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:43:26.144 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:43:26.237 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:43:26.241 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:43:26.276 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:31 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:43:27.274 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:43:27.646 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2023-10-12T18:43:27.906 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2023-10-12T18:43:28.129 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:43:28.131 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================================= 2023-10-12T18:43:28.131 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-10-12T18:43:28.131 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================================= 2023-10-12T18:43:28.131 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:43:28.132 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 27 k 2023-10-12T18:43:28.132 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-10-12T18:43:28.132 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-12T18:43:28.132 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-12T18:43:28.132 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-12T18:43:28.133 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-12T18:43:28.133 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-12T18:43:28.133 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-12T18:43:28.133 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-12T18:43:28.133 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-12T18:43:28.133 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-12T18:43:28.134 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-12T18:43:28.134 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-12T18:43:28.134 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-12T18:43:28.134 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:28.134 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:43:28.135 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================================= 2023-10-12T18:43:28.135 INFO:teuthology.orchestra.run.smithi187.stdout:Install 13 Packages 2023-10-12T18:43:28.135 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:28.136 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 1.9 M 2023-10-12T18:43:28.136 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 24 M 2023-10-12T18:43:28.136 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:43:28.231 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2023-10-12T18:43:28.496 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2023-10-12T18:43:28.544 INFO:teuthology.orchestra.run.smithi187.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 335 kB/s | 16 kB 00:00 2023-10-12T18:43:28.550 INFO:teuthology.orchestra.run.smithi187.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 595 kB/s | 33 kB 00:00 2023-10-12T18:43:28.593 INFO:teuthology.orchestra.run.smithi187.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.6 MB/s | 81 kB 00:00 2023-10-12T18:43:28.606 INFO:teuthology.orchestra.run.smithi187.stdout:(4/13): ceph-mgr-rook-16.2.14-75.g84698e74.el8. 245 kB/s | 27 kB 00:00 2023-10-12T18:43:28.616 INFO:teuthology.orchestra.run.smithi187.stdout:(5/13): python3-rsa-4.9-2.el8.noarch.rpm 2.7 MB/s | 64 kB 00:00 2023-10-12T18:43:28.666 INFO:teuthology.orchestra.run.smithi187.stdout:(6/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 533 kB/s | 27 kB 00:00 2023-10-12T18:43:28.676 INFO:teuthology.orchestra.run.smithi187.stdout:(7/13): python3-websocket-client-0.56.0-5.el8.n 878 kB/s | 61 kB 00:00 2023-10-12T18:43:28.687 INFO:teuthology.orchestra.run.smithi187.stdout:(8/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 6.0 MB/s | 126 kB 00:00 2023-10-12T18:43:28.714 INFO:teuthology.orchestra.run.smithi187.stdout:(9/13): python3-jsonpointer-1.10-11.el8.noarch. 758 kB/s | 20 kB 00:00 2023-10-12T18:43:28.725 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 5/6 2023-10-12T18:43:28.765 INFO:teuthology.orchestra.run.smithi187.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 1.2 MB/s | 110 kB 00:00 2023-10-12T18:43:28.766 INFO:teuthology.orchestra.run.smithi187.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 2.9 MB/s | 155 kB 00:00 2023-10-12T18:43:28.780 INFO:teuthology.orchestra.run.smithi187.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 5.1 MB/s | 1.2 MB 00:00 2023-10-12T18:43:28.828 INFO:teuthology.orchestra.run.smithi187.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 686 kB/s | 43 kB 00:00 2023-10-12T18:43:28.828 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:43:28.829 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.7 MB/s | 1.9 MB 00:00 2023-10-12T18:43:28.834 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:43:28.848 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:43:28.848 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:43:28.856 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-75.g84698e74.el8.noarch 6/6 2023-10-12T18:43:28.939 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:43:28.943 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:43:29.145 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-75.g84698e74.el8.noarch 6/6 2023-10-12T18:43:29.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/6 2023-10-12T18:43:29.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2023-10-12T18:43:29.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2023-10-12T18:43:29.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 4/6 2023-10-12T18:43:29.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 5/6 2023-10-12T18:43:29.455 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:43:29.712 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-12T18:43:29.924 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-10-12T18:43:30.145 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 6/6 2023-10-12T18:43:30.145 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:38 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:30.201 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-10-12T18:43:30.499 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-10-12T18:43:30.732 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-10-12T18:43:30.949 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-10-12T18:43:31.172 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-10-12T18:43:31.424 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-10-12T18:43:31.658 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-12T18:43:31.850 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-10-12T18:43:32.995 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-10-12T18:43:33.308 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-10-12T18:43:33.491 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:16.2.14-75.g84698e74.el8.noarch 13/13 2023-10-12T18:43:33.763 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-75.g84698e74.el8.noarch 13/13 2023-10-12T18:43:33.764 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:16.2.14-75.g84698e74.el8.noarch 1/13 2023-10-12T18:43:33.764 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-10-12T18:43:33.765 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-10-12T18:43:33.765 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-10-12T18:43:33.765 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-10-12T18:43:33.766 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-10-12T18:43:33.766 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-10-12T18:43:33.766 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-10-12T18:43:33.767 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-10-12T18:43:33.767 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-10-12T18:43:33.768 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-12T18:43:33.768 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-10-12T18:43:33.953 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:43:34.051 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:34.052 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:43:34.052 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:43:34.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-apipkg-1.5-6.el8.noarch 2023-10-12T18:43:34.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-12T18:43:34.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-10-12T18:43:34.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-12T18:43:34.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-10-12T18:43:34.053 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:34.053 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:43:34.175 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse 2023-10-12T18:43:34.548 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-10-12T18:43:34.948 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:39 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:43:36.285 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:44 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:38.057 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:43:38.057 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:43:38.058 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-12T18:43:38.058 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:43:38.058 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:43:38.058 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse 2023-10-12T18:43:38.058 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 2:16.2.14-75.g84698e74.el8 ceph 910 k 2023-10-12T18:43:38.059 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-12T18:43:38.059 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-12T18:43:38.059 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:38.059 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:43:38.059 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:43:38.060 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2023-10-12T18:43:38.060 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:38.060 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 993 k 2023-10-12T18:43:38.061 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.8 M 2023-10-12T18:43:38.061 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:43:38.213 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 537 kB/s | 83 kB 00:00 2023-10-12T18:43:38.441 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-fuse-16.2.14-75.g84698e74.el8.x86_6 2.3 MB/s | 910 kB 00:00 2023-10-12T18:43:38.443 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:43:38.444 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.5 MB/s | 993 kB 00:00 2023-10-12T18:43:38.445 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:43:38.469 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:43:38.469 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:43:38.561 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:43:38.565 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:43:39.029 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:43:39.279 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-10-12T18:43:39.438 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:16.2.14-75.g84698e74.el8.x86_64 2/2 2023-10-12T18:43:39.831 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-2:16.2.14-75.g84698e74.el8.x86_64 2/2 2023-10-12T18:43:39.832 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:16.2.14-75.g84698e74.el8.x86_64 1/2 2023-10-12T18:43:40.136 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:43:40.233 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:40.234 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:43:40.234 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:43:40.234 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-12T18:43:40.234 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-12T18:43:40.235 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-12T18:43:40.235 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-12T18:43:40.235 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-12T18:43:40.235 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-12T18:43:40.235 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-12T18:43:40.236 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-12T18:43:40.236 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-12T18:43:40.236 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-12T18:43:40.236 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-12T18:43:40.236 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-12T18:43:40.237 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:40.237 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:43:40.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-cephadm 2023-10-12T18:43:40.775 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:40.849 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-10-12T18:43:40.849 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:48 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:41.988 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:46 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:43:43.294 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:43:43.390 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:43.390 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:43:43.391 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:16.2.14-75.g84698e74.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-10-12T18:43:43.391 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:43.391 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:43:43.534 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel 2023-10-12T18:43:43.909 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:43.943 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:43:43.944 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================== 2023-10-12T18:43:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-10-12T18:43:43.945 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================== 2023-10-12T18:43:43.945 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:43:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:16.2.14-75.g84698e74.el8 ceph-noarch 116 k 2023-10-12T18:43:43.945 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-10-12T18:43:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-10-12T18:43:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-12T18:43:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-10-12T18:43:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-12T18:43:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-10-12T18:43:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:43.947 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:43:43.947 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================== 2023-10-12T18:43:43.947 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2023-10-12T18:43:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:43.948 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 5.8 M 2023-10-12T18:43:43.948 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 24 M 2023-10-12T18:43:43.948 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:43:44.078 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 209 kB/s | 21 kB 00:00 2023-10-12T18:43:44.118 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): python3-remoto-1.2.1-1.el8.noarch.rpm 1.2 MB/s | 49 kB 00:00 2023-10-12T18:43:44.134 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.2 MB/s | 355 kB 00:00 2023-10-12T18:43:44.300 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): python3-babel-2.5.1-5.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2023-10-12T18:43:44.311 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 3.0 MB/s | 538 kB 00:00 2023-10-12T18:43:44.355 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): ceph-mgr-cephadm-16.2.14-75.g84698e74.el 307 kB/s | 116 kB 00:00 2023-10-12T18:43:44.357 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:43:44.358 INFO:teuthology.orchestra.run.smithi187.stdout:Total 14 MB/s | 5.8 MB 00:00 2023-10-12T18:43:44.361 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:43:44.369 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:43:44.370 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:43:44.480 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:43:44.485 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:43:45.514 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:43:45.798 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:53 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:45.905 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2023-10-12T18:43:46.163 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2023-10-12T18:43:46.471 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2023-10-12T18:43:46.720 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2023-10-12T18:43:46.958 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 5/6 2023-10-12T18:43:47.147 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-75.g84698e74.el8.noarch 6/6 2023-10-12T18:43:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-75.g84698e74.el8.noarch 6/6 2023-10-12T18:43:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-75.g84698e74.el8.noarch 1/6 2023-10-12T18:43:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2023-10-12T18:43:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2023-10-12T18:43:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 4/6 2023-10-12T18:43:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 5/6 2023-10-12T18:43:47.656 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:43:47.657 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:43:47.658 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-12T18:43:47.658 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:43:47.658 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:43:47.658 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:16.2.14-75.g84698e74.el8 ceph 131 k 2023-10-12T18:43:47.659 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:47.659 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:43:47.659 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:43:47.659 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-12T18:43:47.659 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:47.660 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 131 k 2023-10-12T18:43:47.660 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 432 k 2023-10-12T18:43:47.660 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:43:48.185 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-16.2.14-75.g84698e74.el8.x86_64. 249 kB/s | 131 kB 00:00 2023-10-12T18:43:48.186 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:43:48.186 INFO:teuthology.orchestra.run.smithi170.stdout:Total 248 kB/s | 131 kB 00:00 2023-10-12T18:43:48.187 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:43:48.193 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:43:48.193 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:43:48.251 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:43:48.255 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:43:48.373 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:43:48.541 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 6/6 2023-10-12T18:43:48.542 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:53 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:43:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:43:48.838 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:43:50.174 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:43:50.175 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:58 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:53.498 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:43:53.597 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:53.598 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:43:53.598 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:16.2.14-75.g84698e74.el8.noarch 2023-10-12T18:43:53.598 INFO:teuthology.orchestra.run.smithi187.stdout: python3-apipkg-1.5-6.el8.noarch 2023-10-12T18:43:53.598 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-12T18:43:53.599 INFO:teuthology.orchestra.run.smithi187.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-10-12T18:43:53.599 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-12T18:43:53.599 INFO:teuthology.orchestra.run.smithi187.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-10-12T18:43:53.599 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:53.599 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:43:53.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-fuse 2023-10-12T18:43:53.722 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:43:53.819 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:53.820 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:43:53.820 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:43:53.820 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:43:53.820 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:43:53.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2 2023-10-12T18:43:54.093 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:54.318 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:55.813 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:00 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:43:55.847 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:03 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:57.162 INFO:teuthology.orchestra.run.smithi170.stdout:Package libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:43:57.640 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:43:57.641 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:43:57.641 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:43:57.652 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:43:57.653 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:43:57.653 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-10-12T18:43:57.653 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:43:57.653 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:43:57.654 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse 2023-10-12T18:43:57.654 INFO:teuthology.orchestra.run.smithi187.stdout: x86_64 2:16.2.14-75.g84698e74.el8 ceph 910 k 2023-10-12T18:43:57.654 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-10-12T18:43:57.654 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-12T18:43:57.655 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:57.655 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:43:57.655 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:43:57.655 INFO:teuthology.orchestra.run.smithi187.stdout:Install 2 Packages 2023-10-12T18:43:57.655 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:43:57.656 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 993 k 2023-10-12T18:43:57.656 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 2.8 M 2023-10-12T18:43:57.656 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:43:57.700 INFO:teuthology.orchestra.run.smithi187.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.8 MB/s | 83 kB 00:00 2023-10-12T18:43:57.772 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs-devel 2023-10-12T18:43:58.143 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:43:58.487 INFO:teuthology.orchestra.run.smithi187.stdout:(2/2): ceph-fuse-16.2.14-75.g84698e74.el8.x86_6 1.1 MB/s | 910 kB 00:00 2023-10-12T18:43:58.488 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:43:58.489 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.2 MB/s | 993 kB 00:00 2023-10-12T18:43:58.490 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:43:58.512 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:43:58.512 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:43:58.595 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:43:58.600 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:43:58.771 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:43:59.012 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-10-12T18:43:59.047 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:07 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:43:59.178 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:16.2.14-75.g84698e74.el8.x86_64 2/2 2023-10-12T18:43:59.588 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-fuse-2:16.2.14-75.g84698e74.el8.x86_64 2/2 2023-10-12T18:43:59.588 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:16.2.14-75.g84698e74.el8.x86_64 1/2 2023-10-12T18:44:00.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-10-12T18:44:00.726 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:05 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:01.047 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:01.048 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:01.048 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-12T18:44:01.048 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:01.049 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:44:01.049 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:16.2.14-75.g84698e74.el8 ceph 25 k 2023-10-12T18:44:01.049 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:01.049 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:44:01.049 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:01.050 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-12T18:44:01.050 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:01.050 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 25 k 2023-10-12T18:44:01.050 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 107 k 2023-10-12T18:44:01.050 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:44:01.397 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs-devel-16.2.14-75.g84698e74.el8.x86_64 73 kB/s | 25 kB 00:00 2023-10-12T18:44:01.398 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:44:01.398 INFO:teuthology.orchestra.run.smithi170.stdout:Total 73 kB/s | 25 kB 00:00 2023-10-12T18:44:01.399 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:44:01.402 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:44:01.403 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:44:01.405 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:44:01.409 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:44:01.466 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:44:01.617 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:01.859 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:02.817 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:02.817 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:10 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:02.894 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:44:02.990 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:02.990 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:44:02.990 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:16.2.14-75.g84698e74.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-10-12T18:44:02.990 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:02.991 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:03.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados-devel 2023-10-12T18:44:03.490 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:04.719 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:09 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:05.223 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:44:05.320 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:05.320 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:44:05.320 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:44:05.321 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:05.321 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:44:05.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2 2023-10-12T18:44:05.812 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:06.589 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:06.590 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:44:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2023-10-12T18:44:06.591 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:44:06.591 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:44:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:16.2.14-75.g84698e74.el8 ceph 131 k 2023-10-12T18:44:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:06.591 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:44:06.592 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:44:06.592 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-10-12T18:44:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:06.592 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 131 k 2023-10-12T18:44:06.592 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 432 k 2023-10-12T18:44:06.593 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:44:07.196 INFO:teuthology.orchestra.run.smithi187.stdout:librados-devel-16.2.14-75.g84698e74.el8.x86_64. 217 kB/s | 131 kB 00:00 2023-10-12T18:44:07.197 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:44:07.197 INFO:teuthology.orchestra.run.smithi187.stdout:Total 216 kB/s | 131 kB 00:00 2023-10-12T18:44:07.198 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:44:07.205 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:44:07.205 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:44:07.260 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:44:07.264 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:44:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:44:07.581 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:15 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:07.601 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:07.875 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:08.886 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:08.998 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:08.998 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:13 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:09.357 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:09.358 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:44:09.358 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:44:09.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1 2023-10-12T18:44:09.852 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:11.277 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:19 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:12.544 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:44:12.602 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:12.642 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:12.643 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:44:12.643 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:44:12.643 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:12.643 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:12.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs2 2023-10-12T18:44:13.072 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:13.073 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:44:13.073 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:44:13.134 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:13.202 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados 2023-10-12T18:44:13.575 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:15.018 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:20 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:15.355 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:23 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:16.397 INFO:teuthology.orchestra.run.smithi187.stdout:Package libcephfs2-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:16.676 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rados-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:16.877 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:16.878 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:44:16.878 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:16.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs-devel 2023-10-12T18:44:17.146 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:17.147 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:44:17.148 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:44:17.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw 2023-10-12T18:44:17.357 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:17.626 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:19.261 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:24 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:19.398 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:27 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:20.709 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rgw-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:21.148 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:21.149 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:44:21.150 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2023-10-12T18:44:21.150 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:44:21.150 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:44:21.150 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:16.2.14-75.g84698e74.el8 ceph 25 k 2023-10-12T18:44:21.151 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:21.151 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:44:21.151 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:44:21.151 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-10-12T18:44:21.151 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:21.152 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 25 k 2023-10-12T18:44:21.152 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 107 k 2023-10-12T18:44:21.152 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:44:21.179 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:21.181 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:44:21.181 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:44:21.441 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs 2023-10-12T18:44:21.544 INFO:teuthology.orchestra.run.smithi187.stdout:libcephfs-devel-16.2.14-75.g84698e74.el8.x86_64 65 kB/s | 25 kB 00:00 2023-10-12T18:44:21.545 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:44:21.546 INFO:teuthology.orchestra.run.smithi187.stdout:Total 64 kB/s | 25 kB 00:00 2023-10-12T18:44:21.546 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:44:21.549 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:44:21.550 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:44:21.552 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:44:21.556 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:44:21.605 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:44:21.748 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:21.814 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:21.991 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:23.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:23.106 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:28 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:23.670 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:31 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:24.975 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-cephfs-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:25.454 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:25.455 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:44:25.455 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:44:25.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd 2023-10-12T18:44:25.948 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:26.569 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:44:26.667 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:26.667 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:44:26.668 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:44:26.668 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:26.668 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:26.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados2 2023-10-12T18:44:27.161 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:27.927 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:35 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:28.976 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:33 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:29.241 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rbd-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:29.712 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:29.713 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-12T18:44:29.713 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:44:29.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse 2023-10-12T18:44:30.199 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:30.362 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:30.831 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:30.832 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:44:30.832 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:30.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librbd1 2023-10-12T18:44:31.312 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:31.856 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:39 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:33.156 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:38 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:33.674 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:33.675 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:33.676 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-10-12T18:44:33.676 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:33.676 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:44:33.676 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:16.2.14-75.g84698e74.el8 ceph 88 k 2023-10-12T18:44:33.676 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:33.677 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:44:33.677 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:33.677 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-12T18:44:33.677 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:33.677 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 88 k 2023-10-12T18:44:33.678 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 240 k 2023-10-12T18:44:33.678 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:44:34.112 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-16.2.14-75.g84698e74.el8.x86_64.rpm 203 kB/s | 88 kB 00:00 2023-10-12T18:44:34.113 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:44:34.114 INFO:teuthology.orchestra.run.smithi170.stdout:Total 202 kB/s | 88 kB 00:00 2023-10-12T18:44:34.114 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:44:34.126 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:44:34.126 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:44:34.183 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:44:34.188 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:44:34.308 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:44:34.511 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:34.512 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:34.782 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:34.979 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:34.980 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:44:34.981 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:35.079 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rados 2023-10-12T18:44:35.451 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:35.818 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:35.819 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:43 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:37.122 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:42 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:37.875 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:44:37.973 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:37.973 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:44:37.973 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:44:37.974 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:37.974 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:44:38.097 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2023-10-12T18:44:38.469 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:38.502 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rados-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:38.974 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:38.975 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:44:38.976 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:39.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rgw 2023-10-12T18:44:39.453 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:39.730 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:47 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:41.322 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:46 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:41.538 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:41.539 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:41.539 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-12T18:44:41.539 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:41.539 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:44:41.539 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:16.2.14-75.g84698e74.el8 ceph 3.7 M 2023-10-12T18:44:41.540 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:41.540 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:44:41.540 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:41.540 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-12T18:44:41.540 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:41.541 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.7 M 2023-10-12T18:44:41.541 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 15 M 2023-10-12T18:44:41.541 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:44:41.948 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-16.2.14-75.g84698e74.el8.x86_64.rpm 9.0 MB/s | 3.7 MB 00:00 2023-10-12T18:44:41.949 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:44:41.949 INFO:teuthology.orchestra.run.smithi170.stdout:Total 9.0 MB/s | 3.7 MB 00:00 2023-10-12T18:44:41.949 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:44:41.963 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:44:41.963 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:44:42.033 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:44:42.038 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:44:42.509 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:44:42.670 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:42.671 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rgw-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:43.087 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:43.144 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:43.145 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:44:43.145 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:43.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-cephfs 2023-10-12T18:44:43.617 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:44.154 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:44.155 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:52 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:45.461 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:50 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:46.826 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-cephfs-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:47.304 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:47.305 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:44:47.306 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:47.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rbd 2023-10-12T18:44:47.781 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:49.260 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:44:49.360 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:49.361 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:44:49.361 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:44:49.361 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:49.361 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:44:49.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd 2023-10-12T18:44:49.614 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:54 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:49.855 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:51.016 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rbd-2:16.2.14-75.g84698e74.el8.x86_64 is already installed. 2023-10-12T18:44:51.455 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:59 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:51.489 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:51.490 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-10-12T18:44:51.490 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:44:51.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-fuse 2023-10-12T18:44:51.969 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:44:53.244 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-12T18:44:53.245 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:53.246 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-10-12T18:44:53.246 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:53.246 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-12T18:44:53.246 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:16.2.14-75.g84698e74.el8 ceph 173 k 2023-10-12T18:44:53.246 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:53.247 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-12T18:44:53.247 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-12T18:44:53.247 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-12T18:44:53.247 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:44:53.247 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 173 k 2023-10-12T18:44:53.248 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 515 k 2023-10-12T18:44:53.248 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-12T18:44:53.392 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:03:58 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:44:53.685 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-16.2.14-75.g84698e74.el8.x86_64.rpm 395 kB/s | 173 kB 00:00 2023-10-12T18:44:53.685 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:44:53.686 INFO:teuthology.orchestra.run.smithi170.stdout:Total 394 kB/s | 173 kB 00:00 2023-10-12T18:44:53.686 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-12T18:44:53.699 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-12T18:44:53.699 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-12T18:44:53.755 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-12T18:44:53.759 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-12T18:44:53.878 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-12T18:44:54.092 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:54.371 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:55.294 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:44:55.295 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:44:55.295 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2023-10-12T18:44:55.295 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:44:55.295 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:44:55.295 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:16.2.14-75.g84698e74.el8 ceph 88 k 2023-10-12T18:44:55.296 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:55.296 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:44:55.296 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:44:55.296 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-10-12T18:44:55.296 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:44:55.297 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 88 k 2023-10-12T18:44:55.297 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 240 k 2023-10-12T18:44:55.297 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:44:55.402 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:55.402 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:04:03 ago on Thu 12 Oct 2023 06:40:52 PM UTC. 2023-10-12T18:44:55.427 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-fuse-16.2.14-75.g84698e74.el8.x86_64.rpm 670 kB/s | 88 kB 00:00 2023-10-12T18:44:55.428 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:44:55.428 INFO:teuthology.orchestra.run.smithi187.stdout:Total 663 kB/s | 88 kB 00:00 2023-10-12T18:44:55.429 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:44:55.440 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:44:55.441 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:44:55.496 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:44:55.501 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:44:55.634 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:44:55.834 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:56.099 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-fuse-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:57.217 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:44:57.217 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:04:02 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:45:01.854 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-10-12T18:45:01.953 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:01.953 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-12T18:45:01.954 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:45:01.954 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:01.954 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-12T18:45:02.067 DEBUG:teuthology.parallel:result is None 2023-10-12T18:45:02.800 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:45:02.897 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:02.897 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:45:02.897 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:45:02.897 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:02.898 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:45:03.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-mirror 2023-10-12T18:45:03.397 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:45:05.240 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:04:10 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:45:07.098 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:45:07.099 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:45:07.100 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2023-10-12T18:45:07.100 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:45:07.100 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:45:07.100 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:16.2.14-75.g84698e74.el8 ceph 3.7 M 2023-10-12T18:45:07.101 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:07.101 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:45:07.101 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:45:07.101 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-10-12T18:45:07.101 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:07.102 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 3.7 M 2023-10-12T18:45:07.102 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 15 M 2023-10-12T18:45:07.102 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:45:08.003 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-mirror-16.2.14-75.g84698e74.el8.x86_64.rpm 4.1 MB/s | 3.7 MB 00:00 2023-10-12T18:45:08.004 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:45:08.004 INFO:teuthology.orchestra.run.smithi187.stdout:Total 4.1 MB/s | 3.7 MB 00:00 2023-10-12T18:45:08.004 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:45:08.019 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:45:08.019 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:45:08.088 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:45:08.093 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:45:08.542 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:45:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:45:09.136 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:45:10.309 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:45:10.310 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:04:15 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:45:14.830 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:45:14.926 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:14.926 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:45:14.927 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:45:14.927 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:14.927 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:45:15.048 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-nbd 2023-10-12T18:45:15.424 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-10-12T18:45:17.308 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:04:22 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:45:19.143 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-12T18:45:19.144 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:45:19.144 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2023-10-12T18:45:19.144 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:45:19.144 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-12T18:45:19.144 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:16.2.14-75.g84698e74.el8 ceph 173 k 2023-10-12T18:45:19.145 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:19.145 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-12T18:45:19.145 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-12T18:45:19.145 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-10-12T18:45:19.145 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:19.146 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 173 k 2023-10-12T18:45:19.146 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 515 k 2023-10-12T18:45:19.146 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-12T18:45:19.407 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-nbd-16.2.14-75.g84698e74.el8.x86_64.rpm 661 kB/s | 173 kB 00:00 2023-10-12T18:45:19.408 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-12T18:45:19.408 INFO:teuthology.orchestra.run.smithi187.stdout:Total 657 kB/s | 173 kB 00:00 2023-10-12T18:45:19.409 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-12T18:45:19.420 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-12T18:45:19.420 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-12T18:45:19.476 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-12T18:45:19.480 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-12T18:45:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-12T18:45:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:45:20.088 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-nbd-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:45:21.211 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:16.2.14-75.g84698e74.el8.x86_64 1/1 2023-10-12T18:45:21.211 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:04:26 ago on Thu 12 Oct 2023 06:40:55 PM UTC. 2023-10-12T18:45:28.175 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-10-12T18:45:28.270 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:28.271 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-12T18:45:28.271 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:16.2.14-75.g84698e74.el8.x86_64 2023-10-12T18:45:28.271 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:28.271 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-12T18:45:28.395 DEBUG:teuthology.parallel:result is None 2023-10-12T18:45:28.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=84698e743199450ee5f6969d22ebe6bed3a68e4a 2023-10-12T18:45:28.542 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-12T18:45:28.566 INFO:teuthology.orchestra.run.smithi170.stdout:16.2.14-75.g84698e74.el8 2023-10-12T18:45:28.567 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-75.g84698e74.el8 2023-10-12T18:45:28.567 INFO:teuthology.task.install:The correct ceph version 16.2.14-75.g84698e74 is installed. 2023-10-12T18:45:28.567 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=84698e743199450ee5f6969d22ebe6bed3a68e4a 2023-10-12T18:45:28.700 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-12T18:45:28.720 INFO:teuthology.orchestra.run.smithi187.stdout:16.2.14-75.g84698e74.el8 2023-10-12T18:45:28.721 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-75.g84698e74.el8 2023-10-12T18:45:28.721 INFO:teuthology.task.install:The correct ceph version 16.2.14-75.g84698e74 is installed. 2023-10-12T18:45:28.721 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-12T18:45:28.722 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:28.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-12T18:45:28.751 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:28.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-12T18:45:28.786 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-12T18:45:28.786 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:28.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2023-10-12T18:45:28.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-12T18:45:28.886 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:28.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2023-10-12T18:45:28.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-12T18:45:28.974 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-12T18:45:28.974 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:28.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-12T18:45:29.001 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-12T18:45:29.065 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:29.066 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-12T18:45:29.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-12T18:45:29.153 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-12T18:45:29.154 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:29.154 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2023-10-12T18:45:29.181 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-12T18:45:29.246 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:29.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2023-10-12T18:45:29.270 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-12T18:45:29.334 INFO:teuthology.run_tasks:Running task ceph... 2023-10-12T18:45:29.395 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-12T18:45:29.396 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2023-10-12T18:45:29.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /var/log/ceph 2023-10-12T18:45:29.422 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-12T18:45:29.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-12T18:45:29.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-12T18:45:29.488 INFO:tasks.ceph:Creating extra log directories... 2023-10-12T18:45:29.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-12T18:45:29.530 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-12T18:45:29.563 INFO:tasks.ceph:Creating ceph cluster c1... 2023-10-12T18:45:29.564 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-12T18:45:29.564 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584', 'branch': 'wip-yuri2-testing-2023-10-11-0853-pacific', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7423584', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2023-10-11-0853-pacific'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-qa-63004-pacific', 'sha1': 'e5d3c6c93855bbe531cbf7186fc3967731bde287'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 4772, 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-qa-63004-pacific', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': 'e5d3c6c93855bbe531cbf7186fc3967731bde287', 'targets': {'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1m8XjWbqD1iXBJEhpX/v8gnKr8KzzzsWDDpPq6Zl+EF15uFoyzlAjob/fRe4yJl3EC86XUqN4NQNwEQD1PzoY=', 'smithi187.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKwWJMgAdGT2iiMDleMD67QKtvCoO64aQexf5vcn57O0419KVr8ejNnxld6r3GrUe8l2tWI+ssJ7+UJv7eJPIME='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-12_17:57:07', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-12T18:45:29.565 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-12T18:45:29.596 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-12T18:45:29.610 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2023-10-12T18:45:29.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/run/ceph 2023-10-12T18:45:29.674 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:29.675 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2023-10-12T18:45:29.728 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-12T18:45:29.729 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2023-10-12T18:45:29.783 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-12T18:45:29.784 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T18:45:29.784 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 6h/6d Inode: 222315 Links: 1 2023-10-12T18:45:29.784 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T18:45:29.784 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T18:45:29.784 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-12 18:44:42.848259446 +0000 2023-10-12T18:45:29.785 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-12 18:40:06.376737584 +0000 2023-10-12T18:45:29.785 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-12 18:40:06.376737584 +0000 2023-10-12T18:45:29.785 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-10-12T18:45:29.785 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-12T18:45:29.846 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-12T18:45:29.847 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-12T18:45:29.847 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000119811 s, 4.3 MB/s 2023-10-12T18:45:29.849 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-12T18:45:29.904 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2023-10-12T18:45:29.959 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-12T18:45:29.959 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T18:45:29.960 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 6h/6d Inode: 218796 Links: 1 2023-10-12T18:45:29.960 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T18:45:29.960 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T18:45:29.960 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-12 18:44:42.848259446 +0000 2023-10-12T18:45:29.960 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-12 18:40:06.868734906 +0000 2023-10-12T18:45:29.961 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-12 18:40:06.868734906 +0000 2023-10-12T18:45:29.961 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-10-12T18:45:29.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-12T18:45:30.022 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-12T18:45:30.023 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-12T18:45:30.023 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000120141 s, 4.3 MB/s 2023-10-12T18:45:30.024 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-12T18:45:30.079 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2023-10-12T18:45:30.134 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-12T18:45:30.134 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T18:45:30.134 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 6h/6d Inode: 217712 Links: 1 2023-10-12T18:45:30.135 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T18:45:30.135 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T18:45:30.135 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-12 18:44:42.848259446 +0000 2023-10-12T18:45:30.135 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-12 18:40:07.352732271 +0000 2023-10-12T18:45:30.135 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-12 18:40:07.352732271 +0000 2023-10-12T18:45:30.136 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-10-12T18:45:30.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-12T18:45:30.197 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-12T18:45:30.197 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-12T18:45:30.197 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000105602 s, 4.8 MB/s 2023-10-12T18:45:30.198 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-12T18:45:30.253 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2023-10-12T18:45:30.308 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-12T18:45:30.309 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T18:45:30.309 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 6h/6d Inode: 220003 Links: 1 2023-10-12T18:45:30.309 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T18:45:30.309 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T18:45:30.309 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-12 18:44:42.848259446 +0000 2023-10-12T18:45:30.310 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-12 18:40:07.853729543 +0000 2023-10-12T18:45:30.310 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-12 18:40:07.853729543 +0000 2023-10-12T18:45:30.310 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-10-12T18:45:30.310 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-12T18:45:30.371 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-12T18:45:30.371 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-12T18:45:30.371 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000120943 s, 4.2 MB/s 2023-10-12T18:45:30.372 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-12T18:45:30.428 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T18:45:30.428 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-12T18:45:30.428 INFO:tasks.ceph:Generating config... 2023-10-12T18:45:30.430 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-12T18:45:30.430 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-12T18:45:30.430 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-12T18:45:30.431 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T18:45:30.431 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-12T18:45:30.431 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-12T18:45:30.432 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-12T18:45:30.432 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-12T18:45:30.432 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-12T18:45:30.433 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-12T18:45:30.433 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-12T18:45:30.433 INFO:tasks.ceph:[client] setuser = ceph 2023-10-12T18:45:30.434 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-12T18:45:30.434 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-12T18:45:30.434 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-12T18:45:30.434 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-12T18:45:30.435 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-12T18:45:30.435 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-12T18:45:30.435 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-12T18:45:30.436 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-12T18:45:30.436 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-12T18:45:30.436 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-12T18:45:30.437 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-12T18:45:30.437 INFO:tasks.ceph:Setting up c1.mon.a... 2023-10-12T18:45:30.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-10-12T18:45:30.538 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/c1.keyring 2023-10-12T18:45:30.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-10-12T18:45:30.578 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-12T18:45:30.641 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.170')] 2023-10-12T18:45:30.641 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.170'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-12T18:45:30.642 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:30.643 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-12T18:45:30.696 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.170 --print /home/ubuntu/cephtest/c1.monmap 2023-10-12T18:45:30.769 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:45:30.770 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:45:30.770 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-10-12T18:45:30.770 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: generated fsid 1d86b02b-9ae0-4b5a-846b-2e446987469a 2023-10-12T18:45:30.771 INFO:teuthology.orchestra.run.smithi170.stdout:epoch 0 2023-10-12T18:45:30.771 INFO:teuthology.orchestra.run.smithi170.stdout:fsid 1d86b02b-9ae0-4b5a-846b-2e446987469a 2023-10-12T18:45:30.771 INFO:teuthology.orchestra.run.smithi170.stdout:last_changed 2023-10-12T18:45:30.769999+0000 2023-10-12T18:45:30.771 INFO:teuthology.orchestra.run.smithi170.stdout:created 2023-10-12T18:45:30.769999+0000 2023-10-12T18:45:30.771 INFO:teuthology.orchestra.run.smithi170.stdout:min_mon_release 0 (unknown) 2023-10-12T18:45:30.771 INFO:teuthology.orchestra.run.smithi170.stdout:election_strategy: 1 2023-10-12T18:45:30.772 INFO:teuthology.orchestra.run.smithi170.stdout:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-12T18:45:30.772 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-10-12T18:45:30.773 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-12T18:45:30.828 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 1d86b02b-9ae0-4b5a-846b-2e446987469a... 2023-10-12T18:45:30.828 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-12T18:45:30.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-12T18:45:30.909 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2023-10-12T18:45:30.909 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2023-10-12T18:45:30.909 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-12T18:45:30.909 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2023-10-12T18:45:30.910 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.910 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2023-10-12T18:45:30.910 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.910 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2023-10-12T18:45:30.910 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.911 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2023-10-12T18:45:30.911 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2023-10-12T18:45:30.911 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.911 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2023-10-12T18:45:30.911 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2023-10-12T18:45:30.911 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.912 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-12T18:45:30.912 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2023-10-12T18:45:30.912 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.912 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2023-10-12T18:45:30.912 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-12T18:45:30.913 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.913 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2023-10-12T18:45:30.913 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.913 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2023-10-12T18:45:30.913 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2023-10-12T18:45:30.914 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2023-10-12T18:45:30.914 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2023-10-12T18:45:30.914 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2023-10-12T18:45:30.914 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2023-10-12T18:45:30.914 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2023-10-12T18:45:30.914 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-12T18:45:30.915 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-12T18:45:30.915 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2023-10-12T18:45:30.915 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.915 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-10-12T18:45:30.915 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.916 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2023-10-12T18:45:30.916 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.916 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2023-10-12T18:45:30.916 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.917 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2023-10-12T18:45:30.917 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2023-10-12T18:45:30.917 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2023-10-12T18:45:30.917 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = 172.21.15.170 2023-10-12T18:45:30.917 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = 1d86b02b-9ae0-4b5a-846b-2e446987469a 2023-10-12T18:45:30.917 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.918 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2023-10-12T18:45:30.918 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2023-10-12T18:45:30.918 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.918 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2023-10-12T18:45:30.918 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2023-10-12T18:45:30.919 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.919 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2023-10-12T18:45:30.919 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2023-10-12T18:45:30.919 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.919 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2023-10-12T18:45:30.920 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2023-10-12T18:45:30.920 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2023-10-12T18:45:30.920 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.920 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2023-10-12T18:45:30.920 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2023-10-12T18:45:30.920 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.921 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2023-10-12T18:45:30.921 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.921 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2023-10-12T18:45:30.921 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.921 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2023-10-12T18:45:30.922 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.922 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2023-10-12T18:45:30.922 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2023-10-12T18:45:30.922 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.922 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2023-10-12T18:45:30.923 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2023-10-12T18:45:30.923 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-12T18:45:30.923 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2023-10-12T18:45:30.923 INFO:teuthology.orchestra.run.smithi170.stdout: rocksdb delete range threshold = 0 2023-10-12T18:45:30.923 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.923 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2023-10-12T18:45:30.924 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-12T18:45:30.924 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2023-10-12T18:45:30.924 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-12T18:45:30.924 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-12T18:45:30.924 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-12T18:45:30.925 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-12T18:45:30.925 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2023-10-12T18:45:30.925 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.925 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2023-10-12T18:45:30.925 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-12T18:45:30.925 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-12T18:45:30.926 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2023-10-12T18:45:30.926 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-12T18:45:30.926 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2023-10-12T18:45:30.926 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2023-10-12T18:45:30.926 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-12T18:45:30.927 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2023-10-12T18:45:30.927 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2023-10-12T18:45:30.927 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-12T18:45:30.927 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.927 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-12T18:45:30.928 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2023-10-12T18:45:30.928 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2023-10-12T18:45:30.928 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.928 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2023-10-12T18:45:30.928 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-12T18:45:30.928 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-12T18:45:30.929 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:30.929 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2023-10-12T18:45:30.929 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2023-10-12T18:45:30.929 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2023-10-12T18:45:30.929 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2023-10-12T18:45:30.930 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-12T18:45:30.930 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-12T18:45:30.930 INFO:teuthology.orchestra.run.smithi170.stdout: debug rgw = 20 2023-10-12T18:45:30.930 INFO:teuthology.orchestra.run.smithi170.stdout: rgw crypt require ssl = False 2023-10-12T18:45:30.930 INFO:teuthology.orchestra.run.smithi170.stdout: rgw crypt s3 kms backend = testing 2023-10-12T18:45:30.931 INFO:teuthology.orchestra.run.smithi170.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T18:45:30.931 INFO:teuthology.orchestra.run.smithi170.stdout: rgw curl low speed time = 300 2023-10-12T18:45:30.931 INFO:teuthology.orchestra.run.smithi170.stdout: rgw data log num shards = 4 2023-10-12T18:45:30.931 INFO:teuthology.orchestra.run.smithi170.stdout: rgw md log max shards = 4 2023-10-12T18:45:30.931 INFO:teuthology.orchestra.run.smithi170.stdout: rgw sync log trim interval = 0 2023-10-12T18:45:30.931 INFO:teuthology.orchestra.run.smithi170.stdout: rgw sync meta inject err probability = 0.1 2023-10-12T18:45:30.932 INFO:teuthology.orchestra.run.smithi170.stdout: rgw sync obj etag verify = True 2023-10-12T18:45:30.932 INFO:teuthology.orchestra.run.smithi170.stdout: setgroup = ceph 2023-10-12T18:45:30.932 INFO:teuthology.orchestra.run.smithi170.stdout: setuser = ceph 2023-10-12T18:45:30.932 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2023-10-12T18:45:30.932 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2023-10-12T18:45:30.933 INFO:teuthology.orchestra.run.smithi170.stdout: osd_max_omap_entries_per_request = 10 2023-10-12T18:45:30.933 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2023-10-12T18:45:30.933 INFO:teuthology.orchestra.run.smithi170.stdout: osd_max_omap_entries_per_request = 1000 2023-10-12T18:45:30.933 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2023-10-12T18:45:30.933 INFO:teuthology.orchestra.run.smithi170.stdout: osd_max_omap_entries_per_request = 10000 2023-10-12T18:45:30.935 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2023-10-12T18:45:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2023-10-12T18:45:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-12T18:45:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2023-10-12T18:45:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2023-10-12T18:45:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2023-10-12T18:45:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2023-10-12T18:45:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2023-10-12T18:45:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2023-10-12T18:45:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2023-10-12T18:45:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-12T18:45:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2023-10-12T18:45:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2023-10-12T18:45:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-12T18:45:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2023-10-12T18:45:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2023-10-12T18:45:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2023-10-12T18:45:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2023-10-12T18:45:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2023-10-12T18:45:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2023-10-12T18:45:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2023-10-12T18:45:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2023-10-12T18:45:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-12T18:45:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-12T18:45:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2023-10-12T18:45:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-10-12T18:45:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2023-10-12T18:45:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2023-10-12T18:45:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2023-10-12T18:45:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2023-10-12T18:45:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2023-10-12T18:45:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = 172.21.15.170 2023-10-12T18:45:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 1d86b02b-9ae0-4b5a-846b-2e446987469a 2023-10-12T18:45:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.943 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2023-10-12T18:45:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2023-10-12T18:45:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2023-10-12T18:45:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2023-10-12T18:45:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2023-10-12T18:45:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2023-10-12T18:45:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2023-10-12T18:45:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2023-10-12T18:45:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2023-10-12T18:45:30.946 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.946 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2023-10-12T18:45:30.946 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2023-10-12T18:45:30.946 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.946 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2023-10-12T18:45:30.947 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.947 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2023-10-12T18:45:30.947 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.947 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2023-10-12T18:45:30.947 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.947 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2023-10-12T18:45:30.948 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2023-10-12T18:45:30.948 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.948 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2023-10-12T18:45:30.948 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2023-10-12T18:45:30.948 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2023-10-12T18:45:30.948 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2023-10-12T18:45:30.949 INFO:teuthology.orchestra.run.smithi187.stdout: rocksdb delete range threshold = 0 2023-10-12T18:45:30.949 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.949 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2023-10-12T18:45:30.949 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2023-10-12T18:45:30.949 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2023-10-12T18:45:30.950 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2023-10-12T18:45:30.950 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2023-10-12T18:45:30.950 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2023-10-12T18:45:30.950 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2023-10-12T18:45:30.950 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2023-10-12T18:45:30.950 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.951 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2023-10-12T18:45:30.951 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2023-10-12T18:45:30.951 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2023-10-12T18:45:30.951 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2023-10-12T18:45:30.951 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2023-10-12T18:45:30.951 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2023-10-12T18:45:30.952 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2023-10-12T18:45:30.952 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2023-10-12T18:45:30.952 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2023-10-12T18:45:30.952 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2023-10-12T18:45:30.952 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2023-10-12T18:45:30.953 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.953 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-12T18:45:30.953 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2023-10-12T18:45:30.953 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2023-10-12T18:45:30.953 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.953 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2023-10-12T18:45:30.954 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-12T18:45:30.954 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-12T18:45:30.954 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:30.954 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2023-10-12T18:45:30.954 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2023-10-12T18:45:30.955 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2023-10-12T18:45:30.955 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2023-10-12T18:45:30.955 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-12T18:45:30.955 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-12T18:45:30.955 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw = 20 2023-10-12T18:45:30.955 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt require ssl = False 2023-10-12T18:45:30.956 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt s3 kms backend = testing 2023-10-12T18:45:30.956 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T18:45:30.956 INFO:teuthology.orchestra.run.smithi187.stdout: rgw curl low speed time = 300 2023-10-12T18:45:30.956 INFO:teuthology.orchestra.run.smithi187.stdout: rgw data log num shards = 4 2023-10-12T18:45:30.956 INFO:teuthology.orchestra.run.smithi187.stdout: rgw md log max shards = 4 2023-10-12T18:45:30.957 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync log trim interval = 0 2023-10-12T18:45:30.957 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync meta inject err probability = 0.1 2023-10-12T18:45:30.957 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync obj etag verify = True 2023-10-12T18:45:30.957 INFO:teuthology.orchestra.run.smithi187.stdout: setgroup = ceph 2023-10-12T18:45:30.957 INFO:teuthology.orchestra.run.smithi187.stdout: setuser = ceph 2023-10-12T18:45:30.958 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2023-10-12T18:45:30.958 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2023-10-12T18:45:30.958 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 10 2023-10-12T18:45:30.958 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2023-10-12T18:45:30.958 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 1000 2023-10-12T18:45:30.958 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2023-10-12T18:45:30.959 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 10000 2023-10-12T18:45:30.959 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-10-12T18:45:30.959 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-10-12T18:45:31.023 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-12T18:45:31.024 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:31.024 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-10-12T18:45:31.039 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:31.039 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-10-12T18:45:31.092 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T18:45:31.093 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:31.093 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/c1.keyring 2023-10-12T18:45:31.094 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-12T18:45:31.166 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:31.166 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-12T18:45:31.221 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T18:45:31.222 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:31.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/c1.keyring 2023-10-12T18:45:31.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-12T18:45:31.259 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:31.260 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-12T18:45:31.315 INFO:tasks.ceph:Setting up mon nodes... 2023-10-12T18:45:31.316 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-12T18:45:31.316 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-10-12T18:45:31.363 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-10-12T18:45:31.365 INFO:tasks.ceph:Setting up mds nodes... 2023-10-12T18:45:31.365 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-12T18:45:31.366 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-10-12T18:45:31.402 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/c1.client.0.keyring 2023-10-12T18:45:31.414 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-10-12T18:45:31.491 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/c1.client.1.keyring 2023-10-12T18:45:31.503 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-12T18:45:31.503 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-12T18:45:31.504 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-10-12T18:45:31.567 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T18:45:31.567 INFO:tasks.ceph:role: c1.osd.0 2023-10-12T18:45:31.568 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T18:45:31.568 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-12T18:45:31.631 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T18:45:31.631 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T18:45:31.631 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T18:45:31.631 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 2023-10-12T18:45:31.632 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T18:45:31.632 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-12T18:45:31.632 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T18:45:31.632 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T18:45:31.632 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T18:45:31.633 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T18:45:31.654 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-12T18:45:31.656 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-12T18:45:31.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-10-12T18:45:31.733 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-10-12T18:45:31.802 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-10-12T18:45:31.866 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T18:45:31.867 INFO:tasks.ceph:role: c1.osd.1 2023-10-12T18:45:31.867 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T18:45:31.867 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-12T18:45:31.930 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T18:45:31.930 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T18:45:31.930 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T18:45:31.931 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 2023-10-12T18:45:31.931 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T18:45:31.931 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-12T18:45:31.931 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T18:45:31.931 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T18:45:31.931 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T18:45:31.932 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T18:45:31.952 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-12T18:45:31.954 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-12T18:45:31.954 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-10-12T18:45:32.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-10-12T18:45:32.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-10-12T18:45:32.163 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T18:45:32.164 INFO:tasks.ceph:role: c1.osd.2 2023-10-12T18:45:32.164 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T18:45:32.164 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-12T18:45:32.227 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T18:45:32.228 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T18:45:32.228 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T18:45:32.228 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 2023-10-12T18:45:32.228 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T18:45:32.228 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-12T18:45:32.229 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T18:45:32.229 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T18:45:32.229 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T18:45:32.229 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T18:45:32.250 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-12T18:45:32.252 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-12T18:45:32.252 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-10-12T18:45:32.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-10-12T18:45:32.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-12T18:45:32.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:32.483+0000 7fadb0c17200 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-10-12T18:45:32.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:32.483+0000 7fadb0c17200 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-10-12T18:45:32.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:32.483+0000 7fadb0c17200 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-12T18:45:32.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:32.483+0000 7fadb0c17200 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-12T18:45:32.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:32.483+0000 7fadb0c17200 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-12T18:45:32.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:32.483+0000 7fadb0c17200 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-10-12T18:45:34.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-10-12T18:45:34.581 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-12T18:45:34.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:34.659+0000 7f1173cf8200 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-10-12T18:45:34.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:34.660+0000 7f1173cf8200 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-10-12T18:45:34.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:34.660+0000 7f1173cf8200 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-12T18:45:34.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:34.660+0000 7f1173cf8200 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-12T18:45:34.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:34.660+0000 7f1173cf8200 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-12T18:45:34.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:34.660+0000 7f1173cf8200 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-10-12T18:45:36.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-10-12T18:45:36.799 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-12T18:45:36.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:36.882+0000 7f8317cd9200 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-10-12T18:45:36.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:36.883+0000 7f8317cd9200 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-10-12T18:45:36.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:36.883+0000 7f8317cd9200 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-12T18:45:36.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:36.883+0000 7f8317cd9200 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-12T18:45:36.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:36.883+0000 7f8317cd9200 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-12T18:45:36.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:45:36.883+0000 7f8317cd9200 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-10-12T18:45:38.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-10-12T18:45:38.983 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-12T18:45:38.984 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:38.984 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-10-12T18:45:39.048 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:39.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-10-12T18:45:39.113 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:39.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-10-12T18:45:39.178 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:39.178 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-10-12T18:45:39.241 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:39.242 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-10-12T18:45:39.296 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:39.296 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-10-12T18:45:39.350 INFO:tasks.ceph:Adding keys to all mons... 2023-10-12T18:45:39.350 DEBUG:teuthology.orchestra.run.smithi170:> sudo tee -a /etc/ceph/c1.keyring 2023-10-12T18:45:39.412 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.x] 2023-10-12T18:45:39.412 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDLPihl9YikFRAAF96FIUA3+ZtY0hYQzf19aA== 2023-10-12T18:45:39.413 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2023-10-12T18:45:39.413 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDMPihlY87jHBAAaO1PGLHb768gb7T6bzw1cw== 2023-10-12T18:45:39.413 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2023-10-12T18:45:39.413 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOPihlrRhuJxAAm7p6+P8ApkSYtvfNJA0umQ== 2023-10-12T18:45:39.414 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2023-10-12T18:45:39.414 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDQPihlRCi4NBAArHYKmpNlcLFz5OIouI1KSA== 2023-10-12T18:45:39.414 INFO:teuthology.orchestra.run.smithi170.stdout:[client.0] 2023-10-12T18:45:39.414 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDLPihlAUT6FxAAidufjbx+9aZV5w+zK8j4xA== 2023-10-12T18:45:39.415 INFO:teuthology.orchestra.run.smithi170.stdout:[client.1] 2023-10-12T18:45:39.415 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDLPihlfN9HHRAAfgGX5uH4dsknhjJu1QY3mQ== 2023-10-12T18:45:39.416 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-12T18:45:39.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T18:45:39.548 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T18:45:39.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T18:45:39.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-12T18:45:39.690 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-12T18:45:39.738 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-12T18:45:39.739 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-10-12T18:45:39.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-10-12T18:45:39.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-10-12T18:45:39.878 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-10-12T18:45:39.932 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-10-12T18:45:39.933 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-10-12T18:45:39.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-10-12T18:45:39.975 INFO:tasks.ceph.c1.mon.a:Started 2023-10-12T18:45:39.975 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-10-12T18:45:39.976 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-10-12T18:45:39.976 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-10-12T18:45:39.978 INFO:tasks.ceph.c1.mgr.x:Started 2023-10-12T18:45:39.978 INFO:tasks.ceph:Setting crush tunables to default 2023-10-12T18:45:39.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster c1 osd crush tunables default 2023-10-12T18:45:40.173 INFO:teuthology.orchestra.run.smithi170.stderr:adjusted tunables profile to default 2023-10-12T18:45:40.183 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-10-12T18:45:40.184 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:40.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-10-12T18:45:40.208 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:40.208 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-10-12T18:45:40.273 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:40.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-10-12T18:45:40.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster c1 osd new 7b28e758-0399-4d53-a70f-db692b665257 0 2023-10-12T18:45:40.382 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:40.380+0000 7f18b6576680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-12T18:45:40.454 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:40.453+0000 7f18b6576680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-12T18:45:40.582 INFO:teuthology.orchestra.run.smithi170.stdout:0 2023-10-12T18:45:40.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster c1 osd new 42fabdc9-4454-4f08-b147-fd3b5c276f02 1 2023-10-12T18:45:40.837 INFO:teuthology.orchestra.run.smithi170.stdout:1 2023-10-12T18:45:40.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster c1 osd new abf486db-9793-43f5-af5f-5d4fea2a1a1f 2 2023-10-12T18:45:40.938 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:40.936+0000 7f18b6576680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-12T18:45:41.052 INFO:teuthology.orchestra.run.smithi170.stdout:2 2023-10-12T18:45:41.064 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-10-12T18:45:41.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-10-12T18:45:41.067 INFO:tasks.ceph.c1.osd.0:Started 2023-10-12T18:45:41.067 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-10-12T18:45:41.067 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-10-12T18:45:41.070 INFO:tasks.ceph.c1.osd.1:Started 2023-10-12T18:45:41.070 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-10-12T18:45:41.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-10-12T18:45:41.072 INFO:tasks.ceph.c1.osd.2:Started 2023-10-12T18:45:41.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-12T18:45:41.268 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:41.269 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":5,"fsid":"1d86b02b-9ae0-4b5a-846b-2e446987469a","created":"2023-10-12T18:45:40.082660+0000","modified":"2023-10-12T18:45:41.050780+0000","last_up_change":"0.000000","last_in_change":"2023-10-12T18:45:41.050780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7b28e758-0399-4d53-a70f-db692b665257","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42fabdc9-4454-4f08-b147-fd3b5c276f02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"abf486db-9793-43f5-af5f-5d4fea2a1a1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:45:41.281 INFO:tasks.ceph.ceph_manager.c1:[] 2023-10-12T18:45:41.281 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-12T18:45:41.584 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-12T18:45:41.690 INFO:tasks.ceph.c1.osd.0.smithi170.stderr:2023-10-12T18:45:41.688+0000 7fc5abbbc200 -1 Falling back to public interface 2023-10-12T18:45:41.697 INFO:tasks.ceph.c1.osd.1.smithi170.stderr:2023-10-12T18:45:41.695+0000 7f8157cf6200 -1 Falling back to public interface 2023-10-12T18:45:41.699 INFO:tasks.ceph.c1.osd.2.smithi170.stderr:2023-10-12T18:45:41.697+0000 7fc16f870200 -1 Falling back to public interface 2023-10-12T18:45:41.701 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:41.699+0000 7f18b6576680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-12T18:45:41.771 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:41.769+0000 7f18b6576680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-12T18:45:41.788 INFO:teuthology.misc.health.smithi170.stdout: 2023-10-12T18:45:41.788 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":5,"fsid":"1d86b02b-9ae0-4b5a-846b-2e446987469a","created":"2023-10-12T18:45:40.082660+0000","modified":"2023-10-12T18:45:41.050780+0000","last_up_change":"0.000000","last_in_change":"2023-10-12T18:45:41.050780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7b28e758-0399-4d53-a70f-db692b665257","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42fabdc9-4454-4f08-b147-fd3b5c276f02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"abf486db-9793-43f5-af5f-5d4fea2a1a1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:45:41.799 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-12T18:45:41.960 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:41.958+0000 7f18b6576680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-12T18:45:42.397 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:42.395+0000 7f18b6576680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-12T18:45:42.499 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:42.498+0000 7f18b6576680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-12T18:45:42.654 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:42.652+0000 7f18b6576680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-12T18:45:42.731 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:42.729+0000 7f18b6576680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-12T18:45:42.800 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:42.798+0000 7f18b6576680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-12T18:45:43.063 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:43.061+0000 7f18b6576680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-12T18:45:43.266 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:43.265+0000 7f18b6576680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-12T18:45:43.934 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:43.932+0000 7f18b6576680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-12T18:45:44.001 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:44.000+0000 7f18b6576680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-12T18:45:44.079 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:44.077+0000 7f18b6576680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-12T18:45:44.233 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:44.231+0000 7f18b6576680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-12T18:45:44.303 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:44.301+0000 7f18b6576680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-12T18:45:44.417 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:44.416+0000 7f18b6576680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-12T18:45:44.445 INFO:tasks.ceph.c1.osd.2.smithi170.stderr:2023-10-12T18:45:44.443+0000 7fc16f870200 -1 osd.2 0 log_to_monitors {default=true} 2023-10-12T18:45:44.448 INFO:tasks.ceph.c1.osd.0.smithi170.stderr:2023-10-12T18:45:44.446+0000 7fc5abbbc200 -1 osd.0 0 log_to_monitors {default=true} 2023-10-12T18:45:44.458 INFO:tasks.ceph.c1.osd.1.smithi170.stderr:2023-10-12T18:45:44.457+0000 7f8157cf6200 -1 osd.1 0 log_to_monitors {default=true} 2023-10-12T18:45:44.535 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:44.533+0000 7f18b6576680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-12T18:45:44.854 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:44.852+0000 7f18b6576680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-12T18:45:44.924 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:44.923+0000 7f18b6576680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-12T18:45:45.504 INFO:tasks.ceph.c1.osd.2.smithi170.stderr:2023-10-12T18:45:45.502+0000 7fc16622d700 -1 osd.2 0 waiting for initial osdmap 2023-10-12T18:45:45.505 INFO:tasks.ceph.c1.osd.0.smithi170.stderr:2023-10-12T18:45:45.502+0000 7fc5a2579700 -1 osd.0 0 waiting for initial osdmap 2023-10-12T18:45:45.505 INFO:tasks.ceph.c1.osd.1.smithi170.stderr:2023-10-12T18:45:45.503+0000 7f814e6b3700 -1 osd.1 0 waiting for initial osdmap 2023-10-12T18:45:45.506 INFO:tasks.ceph.c1.osd.0.smithi170.stderr:2023-10-12T18:45:45.504+0000 7fc59ecf5700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T18:45:45.506 INFO:tasks.ceph.c1.osd.1.smithi170.stderr:2023-10-12T18:45:45.504+0000 7f814ae2f700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T18:45:45.507 INFO:tasks.ceph.c1.osd.2.smithi170.stderr:2023-10-12T18:45:45.504+0000 7fc1629a9700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T18:45:45.944 INFO:tasks.ceph.c1.mgr.x.smithi170.stderr:2023-10-12T18:45:45.942+0000 7f186c7bb700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-12T18:45:48.103 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-12T18:45:48.387 INFO:teuthology.misc.health.smithi170.stdout: 2023-10-12T18:45:48.387 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":9,"fsid":"1d86b02b-9ae0-4b5a-846b-2e446987469a","created":"2023-10-12T18:45:40.082660+0000","modified":"2023-10-12T18:45:47.505335+0000","last_up_change":"2023-10-12T18:45:46.501810+0000","last_in_change":"2023-10-12T18:45:41.050780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-12T18:45:46.539390+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"7b28e758-0399-4d53-a70f-db692b665257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86172},{"type":"v1","addr":"172.21.15.170:6801","nonce":86172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86172},{"type":"v1","addr":"172.21.15.170:6803","nonce":86172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86172},{"type":"v1","addr":"172.21.15.170:6807","nonce":86172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86172},{"type":"v1","addr":"172.21.15.170:6805","nonce":86172}]},"public_addr":"172.21.15.170:6801/86172","cluster_addr":"172.21.15.170:6803/86172","heartbeat_back_addr":"172.21.15.170:6807/86172","heartbeat_front_addr":"172.21.15.170:6805/86172","state":["exists","up"]},{"osd":1,"uuid":"42fabdc9-4454-4f08-b147-fd3b5c276f02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86187},{"type":"v1","addr":"172.21.15.170:6809","nonce":86187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86187},{"type":"v1","addr":"172.21.15.170:6811","nonce":86187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86187},{"type":"v1","addr":"172.21.15.170:6816","nonce":86187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86187},{"type":"v1","addr":"172.21.15.170:6813","nonce":86187}]},"public_addr":"172.21.15.170:6809/86187","cluster_addr":"172.21.15.170:6811/86187","heartbeat_back_addr":"172.21.15.170:6816/86187","heartbeat_front_addr":"172.21.15.170:6813/86187","state":["exists","up"]},{"osd":2,"uuid":"abf486db-9793-43f5-af5f-5d4fea2a1a1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6815","nonce":86188},{"type":"v1","addr":"172.21.15.170:6817","nonce":86188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86188},{"type":"v1","addr":"172.21.15.170:6819","nonce":86188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86188},{"type":"v1","addr":"172.21.15.170:6823","nonce":86188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86188},{"type":"v1","addr":"172.21.15.170:6821","nonce":86188}]},"public_addr":"172.21.15.170:6817/86188","cluster_addr":"172.21.15.170:6819/86188","heartbeat_back_addr":"172.21.15.170:6823/86188","heartbeat_front_addr":"172.21.15.170:6821/86188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.493754+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.397730+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:45:48.399 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-12T18:45:48.399 INFO:tasks.ceph:Creating RBD pool 2023-10-12T18:45:48.399 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster c1 osd pool create rbd 8 2023-10-12T18:45:49.518 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'rbd' created 2023-10-12T18:45:49.529 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-12T18:45:50.521 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-12T18:45:50.533 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-10-12T18:45:50.533 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-10-12T18:45:50.534 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-10-12T18:45:50.534 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-10-12T18:45:50.535 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-12T18:45:50.866 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:50.877 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":85835},{"type":"v1","addr":"172.21.15.170:6825","nonce":85835}]},"active_addr":"172.21.15.170:6825/85835","active_change":"2023-10-12T18:45:44.937332+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":2571654895}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":4068656948}]}]} 2023-10-12T18:45:50.879 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-10-12T18:45:50.879 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-10-12T18:45:50.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-12T18:45:51.166 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:51.166 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":12,"fsid":"1d86b02b-9ae0-4b5a-846b-2e446987469a","created":"2023-10-12T18:45:40.082660+0000","modified":"2023-10-12T18:45:50.512788+0000","last_up_change":"2023-10-12T18:45:46.501810+0000","last_in_change":"2023-10-12T18:45:41.050780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-12T18:45:46.539390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T18:45:48.714939+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7b28e758-0399-4d53-a70f-db692b665257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86172},{"type":"v1","addr":"172.21.15.170:6801","nonce":86172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86172},{"type":"v1","addr":"172.21.15.170:6803","nonce":86172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86172},{"type":"v1","addr":"172.21.15.170:6807","nonce":86172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86172},{"type":"v1","addr":"172.21.15.170:6805","nonce":86172}]},"public_addr":"172.21.15.170:6801/86172","cluster_addr":"172.21.15.170:6803/86172","heartbeat_back_addr":"172.21.15.170:6807/86172","heartbeat_front_addr":"172.21.15.170:6805/86172","state":["exists","up"]},{"osd":1,"uuid":"42fabdc9-4454-4f08-b147-fd3b5c276f02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86187},{"type":"v1","addr":"172.21.15.170:6809","nonce":86187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86187},{"type":"v1","addr":"172.21.15.170:6811","nonce":86187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86187},{"type":"v1","addr":"172.21.15.170:6816","nonce":86187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86187},{"type":"v1","addr":"172.21.15.170:6813","nonce":86187}]},"public_addr":"172.21.15.170:6809/86187","cluster_addr":"172.21.15.170:6811/86187","heartbeat_back_addr":"172.21.15.170:6816/86187","heartbeat_front_addr":"172.21.15.170:6813/86187","state":["exists","up"]},{"osd":2,"uuid":"abf486db-9793-43f5-af5f-5d4fea2a1a1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6815","nonce":86188},{"type":"v1","addr":"172.21.15.170:6817","nonce":86188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86188},{"type":"v1","addr":"172.21.15.170:6819","nonce":86188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86188},{"type":"v1","addr":"172.21.15.170:6823","nonce":86188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86188},{"type":"v1","addr":"172.21.15.170:6821","nonce":86188}]},"public_addr":"172.21.15.170:6817/86188","cluster_addr":"172.21.15.170:6819/86188","heartbeat_back_addr":"172.21.15.170:6823/86188","heartbeat_front_addr":"172.21.15.170:6821/86188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.442456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.493754+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.397730+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:45:51.179 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-10-12T18:45:51.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-12T18:45:51.466 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:51.466 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":12,"fsid":"1d86b02b-9ae0-4b5a-846b-2e446987469a","created":"2023-10-12T18:45:40.082660+0000","modified":"2023-10-12T18:45:50.512788+0000","last_up_change":"2023-10-12T18:45:46.501810+0000","last_in_change":"2023-10-12T18:45:41.050780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-12T18:45:46.539390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T18:45:48.714939+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7b28e758-0399-4d53-a70f-db692b665257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86172},{"type":"v1","addr":"172.21.15.170:6801","nonce":86172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86172},{"type":"v1","addr":"172.21.15.170:6803","nonce":86172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86172},{"type":"v1","addr":"172.21.15.170:6807","nonce":86172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86172},{"type":"v1","addr":"172.21.15.170:6805","nonce":86172}]},"public_addr":"172.21.15.170:6801/86172","cluster_addr":"172.21.15.170:6803/86172","heartbeat_back_addr":"172.21.15.170:6807/86172","heartbeat_front_addr":"172.21.15.170:6805/86172","state":["exists","up"]},{"osd":1,"uuid":"42fabdc9-4454-4f08-b147-fd3b5c276f02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86187},{"type":"v1","addr":"172.21.15.170:6809","nonce":86187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86187},{"type":"v1","addr":"172.21.15.170:6811","nonce":86187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86187},{"type":"v1","addr":"172.21.15.170:6816","nonce":86187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86187},{"type":"v1","addr":"172.21.15.170:6813","nonce":86187}]},"public_addr":"172.21.15.170:6809/86187","cluster_addr":"172.21.15.170:6811/86187","heartbeat_back_addr":"172.21.15.170:6816/86187","heartbeat_front_addr":"172.21.15.170:6813/86187","state":["exists","up"]},{"osd":2,"uuid":"abf486db-9793-43f5-af5f-5d4fea2a1a1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6815","nonce":86188},{"type":"v1","addr":"172.21.15.170:6817","nonce":86188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86188},{"type":"v1","addr":"172.21.15.170:6819","nonce":86188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86188},{"type":"v1","addr":"172.21.15.170:6823","nonce":86188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86188},{"type":"v1","addr":"172.21.15.170:6821","nonce":86188}]},"public_addr":"172.21.15.170:6817/86188","cluster_addr":"172.21.15.170:6819/86188","heartbeat_back_addr":"172.21.15.170:6823/86188","heartbeat_front_addr":"172.21.15.170:6821/86188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.442456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.493754+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.397730+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:45:51.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-10-12T18:45:51.625 INFO:teuthology.orchestra.run.smithi170.stdout:34359738370 2023-10-12T18:45:51.626 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-10-12T18:45:51.770 INFO:teuthology.orchestra.run.smithi170.stdout:34359738370 2023-10-12T18:45:51.771 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-10-12T18:45:51.915 INFO:teuthology.orchestra.run.smithi170.stdout:34359738370 2023-10-12T18:45:51.916 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-12T18:45:52.201 INFO:teuthology.orchestra.run.smithi170.stdout:34359738369 2023-10-12T18:45:52.215 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2023-10-12T18:45:53.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-12T18:45:53.505 INFO:teuthology.orchestra.run.smithi170.stdout:34359738369 2023-10-12T18:45:53.519 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2023-10-12T18:45:54.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-12T18:45:54.806 INFO:teuthology.orchestra.run.smithi170.stdout:34359738371 2023-10-12T18:45:54.819 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738371 for osd.0 2023-10-12T18:45:54.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-10-12T18:45:55.105 INFO:teuthology.orchestra.run.smithi170.stdout:34359738371 2023-10-12T18:45:55.118 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738371 for osd.1 2023-10-12T18:45:55.119 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-10-12T18:45:55.403 INFO:teuthology.orchestra.run.smithi170.stdout:34359738371 2023-10-12T18:45:55.417 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738371 for osd.2 2023-10-12T18:45:55.417 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-10-12T18:45:55.418 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-12T18:45:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:55.701 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-12T18:45:55.712 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-10-12T18:45:54.942227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681200,"statfs":{"total":322122547200,"available":321209548800,"internally_reserved":0,"allocated":319488,"data_stored":86889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520567+0000","last_change":"2023-10-12T18:45:50.519945+0000","last_active":"2023-10-12T18:45:50.520567+0000","last_peered":"2023-10-12T18:45:50.520567+0000","last_clean":"2023-10-12T18:45:50.520567+0000","last_became_active":"2023-10-12T18:45:50.519711+0000","last_became_peered":"2023-10-12T18:45:50.519711+0000","last_unstale":"2023-10-12T18:45:50.520567+0000","last_undegraded":"2023-10-12T18:45:50.520567+0000","last_fullsized":"2023-10-12T18:45:50.520567+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520650+0000","last_change":"2023-10-12T18:45:50.519983+0000","last_active":"2023-10-12T18:45:50.520650+0000","last_peered":"2023-10-12T18:45:50.520650+0000","last_clean":"2023-10-12T18:45:50.520650+0000","last_became_active":"2023-10-12T18:45:50.519750+0000","last_became_peered":"2023-10-12T18:45:50.519750+0000","last_unstale":"2023-10-12T18:45:50.520650+0000","last_undegraded":"2023-10-12T18:45:50.520650+0000","last_fullsized":"2023-10-12T18:45:50.520650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.521598+0000","last_change":"2023-10-12T18:45:50.521185+0000","last_active":"2023-10-12T18:45:50.521598+0000","last_peered":"2023-10-12T18:45:50.521598+0000","last_clean":"2023-10-12T18:45:50.521598+0000","last_became_active":"2023-10-12T18:45:50.520745+0000","last_became_peered":"2023-10-12T18:45:50.520745+0000","last_unstale":"2023-10-12T18:45:50.521598+0000","last_undegraded":"2023-10-12T18:45:50.521598+0000","last_fullsized":"2023-10-12T18:45:50.521598+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.519206+0000","last_change":"2023-10-12T18:45:50.519206+0000","last_active":"2023-10-12T18:45:50.519206+0000","last_peered":"2023-10-12T18:45:50.519206+0000","last_clean":"2023-10-12T18:45:50.519206+0000","last_became_active":"2023-10-12T18:45:50.519008+0000","last_became_peered":"2023-10-12T18:45:50.519008+0000","last_unstale":"2023-10-12T18:45:50.519206+0000","last_undegraded":"2023-10-12T18:45:50.519206+0000","last_fullsized":"2023-10-12T18:45:50.519206+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.516380+0000","last_change":"2023-10-12T18:45:48.512731+0000","last_active":"2023-10-12T18:45:50.516380+0000","last_peered":"2023-10-12T18:45:50.516380+0000","last_clean":"2023-10-12T18:45:50.516380+0000","last_became_active":"2023-10-12T18:45:48.512502+0000","last_became_peered":"2023-10-12T18:45:48.512502+0000","last_unstale":"2023-10-12T18:45:50.516380+0000","last_undegraded":"2023-10-12T18:45:50.516380+0000","last_fullsized":"2023-10-12T18:45:50.516380+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:47.505335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:47.505335+0000","last_clean_scrub_stamp":"2023-10-12T18:45:47.505335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520140+0000","last_change":"2023-10-12T18:45:50.518492+0000","last_active":"2023-10-12T18:45:50.520140+0000","last_peered":"2023-10-12T18:45:50.520140+0000","last_clean":"2023-10-12T18:45:50.520140+0000","last_became_active":"2023-10-12T18:45:50.518253+0000","last_became_peered":"2023-10-12T18:45:50.518253+0000","last_unstale":"2023-10-12T18:45:50.520140+0000","last_undegraded":"2023-10-12T18:45:50.520140+0000","last_fullsized":"2023-10-12T18:45:50.520140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.518807+0000","last_change":"2023-10-12T18:45:50.518807+0000","last_active":"2023-10-12T18:45:50.518807+0000","last_peered":"2023-10-12T18:45:50.518807+0000","last_clean":"2023-10-12T18:45:50.518807+0000","last_became_active":"2023-10-12T18:45:50.518546+0000","last_became_peered":"2023-10-12T18:45:50.518546+0000","last_unstale":"2023-10-12T18:45:50.518807+0000","last_undegraded":"2023-10-12T18:45:50.518807+0000","last_fullsized":"2023-10-12T18:45:50.518807+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520954+0000","last_change":"2023-10-12T18:45:50.520954+0000","last_active":"2023-10-12T18:45:50.520954+0000","last_peered":"2023-10-12T18:45:50.520954+0000","last_clean":"2023-10-12T18:45:50.520954+0000","last_became_active":"2023-10-12T18:45:50.520695+0000","last_became_peered":"2023-10-12T18:45:50.520695+0000","last_unstale":"2023-10-12T18:45:50.520954+0000","last_undegraded":"2023-10-12T18:45:50.520954+0000","last_fullsized":"2023-10-12T18:45:50.520954+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520676+0000","last_change":"2023-10-12T18:45:50.519980+0000","last_active":"2023-10-12T18:45:50.520676+0000","last_peered":"2023-10-12T18:45:50.520676+0000","last_clean":"2023-10-12T18:45:50.520676+0000","last_became_active":"2023-10-12T18:45:50.519735+0000","last_became_peered":"2023-10-12T18:45:50.519735+0000","last_unstale":"2023-10-12T18:45:50.520676+0000","last_undegraded":"2023-10-12T18:45:50.520676+0000","last_fullsized":"2023-10-12T18:45:50.520676+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-12T18:45:55.713 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-12T18:45:55.995 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:55.997 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-12T18:45:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-10-12T18:45:54.942227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681200,"statfs":{"total":322122547200,"available":321209548800,"internally_reserved":0,"allocated":319488,"data_stored":86889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520567+0000","last_change":"2023-10-12T18:45:50.519945+0000","last_active":"2023-10-12T18:45:50.520567+0000","last_peered":"2023-10-12T18:45:50.520567+0000","last_clean":"2023-10-12T18:45:50.520567+0000","last_became_active":"2023-10-12T18:45:50.519711+0000","last_became_peered":"2023-10-12T18:45:50.519711+0000","last_unstale":"2023-10-12T18:45:50.520567+0000","last_undegraded":"2023-10-12T18:45:50.520567+0000","last_fullsized":"2023-10-12T18:45:50.520567+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520650+0000","last_change":"2023-10-12T18:45:50.519983+0000","last_active":"2023-10-12T18:45:50.520650+0000","last_peered":"2023-10-12T18:45:50.520650+0000","last_clean":"2023-10-12T18:45:50.520650+0000","last_became_active":"2023-10-12T18:45:50.519750+0000","last_became_peered":"2023-10-12T18:45:50.519750+0000","last_unstale":"2023-10-12T18:45:50.520650+0000","last_undegraded":"2023-10-12T18:45:50.520650+0000","last_fullsized":"2023-10-12T18:45:50.520650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.521598+0000","last_change":"2023-10-12T18:45:50.521185+0000","last_active":"2023-10-12T18:45:50.521598+0000","last_peered":"2023-10-12T18:45:50.521598+0000","last_clean":"2023-10-12T18:45:50.521598+0000","last_became_active":"2023-10-12T18:45:50.520745+0000","last_became_peered":"2023-10-12T18:45:50.520745+0000","last_unstale":"2023-10-12T18:45:50.521598+0000","last_undegraded":"2023-10-12T18:45:50.521598+0000","last_fullsized":"2023-10-12T18:45:50.521598+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.519206+0000","last_change":"2023-10-12T18:45:50.519206+0000","last_active":"2023-10-12T18:45:50.519206+0000","last_peered":"2023-10-12T18:45:50.519206+0000","last_clean":"2023-10-12T18:45:50.519206+0000","last_became_active":"2023-10-12T18:45:50.519008+0000","last_became_peered":"2023-10-12T18:45:50.519008+0000","last_unstale":"2023-10-12T18:45:50.519206+0000","last_undegraded":"2023-10-12T18:45:50.519206+0000","last_fullsized":"2023-10-12T18:45:50.519206+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.516380+0000","last_change":"2023-10-12T18:45:48.512731+0000","last_active":"2023-10-12T18:45:50.516380+0000","last_peered":"2023-10-12T18:45:50.516380+0000","last_clean":"2023-10-12T18:45:50.516380+0000","last_became_active":"2023-10-12T18:45:48.512502+0000","last_became_peered":"2023-10-12T18:45:48.512502+0000","last_unstale":"2023-10-12T18:45:50.516380+0000","last_undegraded":"2023-10-12T18:45:50.516380+0000","last_fullsized":"2023-10-12T18:45:50.516380+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:47.505335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:47.505335+0000","last_clean_scrub_stamp":"2023-10-12T18:45:47.505335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520140+0000","last_change":"2023-10-12T18:45:50.518492+0000","last_active":"2023-10-12T18:45:50.520140+0000","last_peered":"2023-10-12T18:45:50.520140+0000","last_clean":"2023-10-12T18:45:50.520140+0000","last_became_active":"2023-10-12T18:45:50.518253+0000","last_became_peered":"2023-10-12T18:45:50.518253+0000","last_unstale":"2023-10-12T18:45:50.520140+0000","last_undegraded":"2023-10-12T18:45:50.520140+0000","last_fullsized":"2023-10-12T18:45:50.520140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.518807+0000","last_change":"2023-10-12T18:45:50.518807+0000","last_active":"2023-10-12T18:45:50.518807+0000","last_peered":"2023-10-12T18:45:50.518807+0000","last_clean":"2023-10-12T18:45:50.518807+0000","last_became_active":"2023-10-12T18:45:50.518546+0000","last_became_peered":"2023-10-12T18:45:50.518546+0000","last_unstale":"2023-10-12T18:45:50.518807+0000","last_undegraded":"2023-10-12T18:45:50.518807+0000","last_fullsized":"2023-10-12T18:45:50.518807+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520954+0000","last_change":"2023-10-12T18:45:50.520954+0000","last_active":"2023-10-12T18:45:50.520954+0000","last_peered":"2023-10-12T18:45:50.520954+0000","last_clean":"2023-10-12T18:45:50.520954+0000","last_became_active":"2023-10-12T18:45:50.520695+0000","last_became_peered":"2023-10-12T18:45:50.520695+0000","last_unstale":"2023-10-12T18:45:50.520954+0000","last_undegraded":"2023-10-12T18:45:50.520954+0000","last_fullsized":"2023-10-12T18:45:50.520954+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:45:50.520676+0000","last_change":"2023-10-12T18:45:50.519980+0000","last_active":"2023-10-12T18:45:50.520676+0000","last_peered":"2023-10-12T18:45:50.520676+0000","last_clean":"2023-10-12T18:45:50.520676+0000","last_became_active":"2023-10-12T18:45:50.519735+0000","last_became_peered":"2023-10-12T18:45:50.519735+0000","last_unstale":"2023-10-12T18:45:50.520676+0000","last_undegraded":"2023-10-12T18:45:50.520676+0000","last_fullsized":"2023-10-12T18:45:50.520676+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:45:49.510888+0000","last_clean_scrub_stamp":"2023-10-12T18:45:49.510888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-12T18:45:56.009 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-10-12T18:45:56.009 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-10-12T18:45:56.009 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-10-12T18:45:56.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-10-12T18:45:56.332 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:56.345 INFO:teuthology.orchestra.run.smithi170.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-12T18:45:56.345 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-10-12T18:45:56.346 INFO:teuthology.run_tasks:Running task ceph... 2023-10-12T18:45:56.356 INFO:tasks.ceph:Creating ceph cluster c2... 2023-10-12T18:45:56.356 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-12T18:45:56.356 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584', 'branch': 'wip-yuri2-testing-2023-10-11-0853-pacific', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7423584', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2023-10-11-0853-pacific'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-qa-63004-pacific', 'sha1': 'e5d3c6c93855bbe531cbf7186fc3967731bde287'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 4772, 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-qa-63004-pacific', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': 'e5d3c6c93855bbe531cbf7186fc3967731bde287', 'targets': {'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1m8XjWbqD1iXBJEhpX/v8gnKr8KzzzsWDDpPq6Zl+EF15uFoyzlAjob/fRe4yJl3EC86XUqN4NQNwEQD1PzoY=', 'smithi187.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKwWJMgAdGT2iiMDleMD67QKtvCoO64aQexf5vcn57O0419KVr8ejNnxld6r3GrUe8l2tWI+ssJ7+UJv7eJPIME='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '84698e743199450ee5f6969d22ebe6bed3a68e4a', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-12_17:57:07', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-12T18:45:56.357 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-12T18:45:56.359 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-12T18:45:56.376 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2023-10-12T18:45:56.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/run/ceph 2023-10-12T18:45:56.444 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:56.444 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2023-10-12T18:45:56.501 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-12T18:45:56.501 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2023-10-12T18:45:56.557 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-12T18:45:56.557 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T18:45:56.558 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 220600 Links: 1 2023-10-12T18:45:56.558 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T18:45:56.558 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T18:45:56.558 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-10-12 18:45:08.899823835 +0000 2023-10-12T18:45:56.558 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-10-12 18:40:07.879277252 +0000 2023-10-12T18:45:56.559 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-10-12 18:40:07.879277252 +0000 2023-10-12T18:45:56.559 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2023-10-12T18:45:56.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-12T18:45:56.621 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-10-12T18:45:56.622 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-10-12T18:45:56.622 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000262677 s, 1.9 MB/s 2023-10-12T18:45:56.623 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-12T18:45:56.680 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2023-10-12T18:45:56.736 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-12T18:45:56.736 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T18:45:56.736 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 216049 Links: 1 2023-10-12T18:45:56.736 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T18:45:56.737 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T18:45:56.737 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-10-12 18:45:08.899823835 +0000 2023-10-12T18:45:56.737 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-10-12 18:40:08.380269567 +0000 2023-10-12T18:45:56.737 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-10-12 18:40:08.380269567 +0000 2023-10-12T18:45:56.737 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2023-10-12T18:45:56.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-12T18:45:56.800 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-10-12T18:45:56.800 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-10-12T18:45:56.801 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000252601 s, 2.0 MB/s 2023-10-12T18:45:56.801 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-12T18:45:56.858 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2023-10-12T18:45:56.913 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-12T18:45:56.913 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T18:45:56.914 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 220887 Links: 1 2023-10-12T18:45:56.914 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T18:45:56.914 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T18:45:56.914 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-10-12 18:45:08.899823835 +0000 2023-10-12T18:45:56.914 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-10-12 18:40:08.888261775 +0000 2023-10-12T18:45:56.915 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-10-12 18:40:08.888261775 +0000 2023-10-12T18:45:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2023-10-12T18:45:56.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-12T18:45:56.976 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-10-12T18:45:56.977 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-10-12T18:45:56.977 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000222234 s, 2.3 MB/s 2023-10-12T18:45:56.978 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-12T18:45:57.033 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2023-10-12T18:45:57.089 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-12T18:45:57.089 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T18:45:57.089 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 221687 Links: 1 2023-10-12T18:45:57.090 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T18:45:57.090 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T18:45:57.090 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-10-12 18:45:08.899823835 +0000 2023-10-12T18:45:57.090 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-10-12 18:40:09.396253982 +0000 2023-10-12T18:45:57.090 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-10-12 18:40:09.396253982 +0000 2023-10-12T18:45:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2023-10-12T18:45:57.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-12T18:45:57.152 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-10-12T18:45:57.152 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-10-12T18:45:57.152 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000195013 s, 2.6 MB/s 2023-10-12T18:45:57.154 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-12T18:45:57.209 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T18:45:57.210 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-12T18:45:57.210 INFO:tasks.ceph:Generating config... 2023-10-12T18:45:57.211 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-12T18:45:57.212 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-12T18:45:57.212 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-12T18:45:57.212 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T18:45:57.213 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-12T18:45:57.213 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-12T18:45:57.213 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-12T18:45:57.213 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-12T18:45:57.214 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-12T18:45:57.214 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-12T18:45:57.214 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-12T18:45:57.214 INFO:tasks.ceph:[client] setuser = ceph 2023-10-12T18:45:57.215 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-12T18:45:57.215 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-12T18:45:57.215 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-12T18:45:57.215 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-12T18:45:57.216 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-12T18:45:57.216 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-12T18:45:57.216 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-12T18:45:57.216 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-12T18:45:57.217 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-12T18:45:57.217 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-12T18:45:57.217 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-12T18:45:57.217 INFO:tasks.ceph:Setting up c2.mon.a... 2023-10-12T18:45:57.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-10-12T18:45:57.321 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/c2.keyring 2023-10-12T18:45:57.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-10-12T18:45:57.361 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-12T18:45:57.426 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.187')] 2023-10-12T18:45:57.427 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.187'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-12T18:45:57.427 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:57.428 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-12T18:45:57.481 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.187 --print /home/ubuntu/cephtest/c2.monmap 2023-10-12T18:45:57.555 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:45:57.556 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:45:57.557 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-10-12T18:45:57.557 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: generated fsid 171d6df1-cef5-40ec-977b-bbc477f9da52 2023-10-12T18:45:57.557 INFO:teuthology.orchestra.run.smithi187.stdout:epoch 0 2023-10-12T18:45:57.557 INFO:teuthology.orchestra.run.smithi187.stdout:fsid 171d6df1-cef5-40ec-977b-bbc477f9da52 2023-10-12T18:45:57.558 INFO:teuthology.orchestra.run.smithi187.stdout:last_changed 2023-10-12T18:45:57.555885+0000 2023-10-12T18:45:57.558 INFO:teuthology.orchestra.run.smithi187.stdout:created 2023-10-12T18:45:57.555885+0000 2023-10-12T18:45:57.558 INFO:teuthology.orchestra.run.smithi187.stdout:min_mon_release 0 (unknown) 2023-10-12T18:45:57.558 INFO:teuthology.orchestra.run.smithi187.stdout:election_strategy: 1 2023-10-12T18:45:57.558 INFO:teuthology.orchestra.run.smithi187.stdout:0: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.a 2023-10-12T18:45:57.559 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-10-12T18:45:57.559 DEBUG:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-12T18:45:57.612 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 171d6df1-cef5-40ec-977b-bbc477f9da52... 2023-10-12T18:45:57.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-12T18:45:57.615 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-12T18:45:57.657 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2023-10-12T18:45:57.657 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2023-10-12T18:45:57.657 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-12T18:45:57.658 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2023-10-12T18:45:57.658 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.658 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2023-10-12T18:45:57.658 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.659 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2023-10-12T18:45:57.659 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.659 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2023-10-12T18:45:57.659 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2023-10-12T18:45:57.659 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.659 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2023-10-12T18:45:57.660 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2023-10-12T18:45:57.660 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.660 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-12T18:45:57.660 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2023-10-12T18:45:57.660 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.661 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2023-10-12T18:45:57.661 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-12T18:45:57.661 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.661 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2023-10-12T18:45:57.661 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.662 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2023-10-12T18:45:57.662 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2023-10-12T18:45:57.662 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2023-10-12T18:45:57.663 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2023-10-12T18:45:57.663 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2023-10-12T18:45:57.663 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2023-10-12T18:45:57.663 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2023-10-12T18:45:57.663 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-12T18:45:57.663 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-12T18:45:57.664 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2023-10-12T18:45:57.664 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.664 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-10-12T18:45:57.664 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.664 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2023-10-12T18:45:57.665 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.665 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2023-10-12T18:45:57.665 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.665 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2023-10-12T18:45:57.665 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2023-10-12T18:45:57.666 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2023-10-12T18:45:57.666 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = 172.21.15.187 2023-10-12T18:45:57.666 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = 171d6df1-cef5-40ec-977b-bbc477f9da52 2023-10-12T18:45:57.666 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.666 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2023-10-12T18:45:57.666 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2023-10-12T18:45:57.667 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.667 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2023-10-12T18:45:57.667 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2023-10-12T18:45:57.667 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.667 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2023-10-12T18:45:57.668 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2023-10-12T18:45:57.668 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.668 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2023-10-12T18:45:57.668 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2023-10-12T18:45:57.668 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2023-10-12T18:45:57.668 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.669 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2023-10-12T18:45:57.669 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2023-10-12T18:45:57.669 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.669 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2023-10-12T18:45:57.669 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.670 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2023-10-12T18:45:57.670 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.670 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2023-10-12T18:45:57.670 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.670 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2023-10-12T18:45:57.671 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2023-10-12T18:45:57.671 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.671 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2023-10-12T18:45:57.671 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2023-10-12T18:45:57.671 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-12T18:45:57.672 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2023-10-12T18:45:57.672 INFO:teuthology.orchestra.run.smithi170.stdout: rocksdb delete range threshold = 0 2023-10-12T18:45:57.672 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.672 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2023-10-12T18:45:57.673 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-12T18:45:57.673 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2023-10-12T18:45:57.673 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-12T18:45:57.673 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-12T18:45:57.673 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-12T18:45:57.674 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-12T18:45:57.674 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2023-10-12T18:45:57.674 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.674 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2023-10-12T18:45:57.675 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-12T18:45:57.675 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-12T18:45:57.675 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2023-10-12T18:45:57.675 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-12T18:45:57.675 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2023-10-12T18:45:57.676 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2023-10-12T18:45:57.676 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-12T18:45:57.676 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2023-10-12T18:45:57.676 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2023-10-12T18:45:57.677 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-12T18:45:57.677 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.677 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-12T18:45:57.677 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2023-10-12T18:45:57.677 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2023-10-12T18:45:57.678 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.678 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2023-10-12T18:45:57.678 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-12T18:45:57.678 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-12T18:45:57.678 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:45:57.679 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2023-10-12T18:45:57.679 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2023-10-12T18:45:57.679 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2023-10-12T18:45:57.679 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2023-10-12T18:45:57.679 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-12T18:45:57.679 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-12T18:45:57.680 INFO:teuthology.orchestra.run.smithi170.stdout: debug rgw = 20 2023-10-12T18:45:57.680 INFO:teuthology.orchestra.run.smithi170.stdout: rgw crypt require ssl = False 2023-10-12T18:45:57.680 INFO:teuthology.orchestra.run.smithi170.stdout: rgw crypt s3 kms backend = testing 2023-10-12T18:45:57.680 INFO:teuthology.orchestra.run.smithi170.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T18:45:57.680 INFO:teuthology.orchestra.run.smithi170.stdout: rgw curl low speed time = 300 2023-10-12T18:45:57.681 INFO:teuthology.orchestra.run.smithi170.stdout: rgw data log num shards = 4 2023-10-12T18:45:57.681 INFO:teuthology.orchestra.run.smithi170.stdout: rgw md log max shards = 4 2023-10-12T18:45:57.681 INFO:teuthology.orchestra.run.smithi170.stdout: rgw sync log trim interval = 0 2023-10-12T18:45:57.681 INFO:teuthology.orchestra.run.smithi170.stdout: rgw sync meta inject err probability = 0.1 2023-10-12T18:45:57.681 INFO:teuthology.orchestra.run.smithi170.stdout: rgw sync obj etag verify = True 2023-10-12T18:45:57.682 INFO:teuthology.orchestra.run.smithi170.stdout: setgroup = ceph 2023-10-12T18:45:57.682 INFO:teuthology.orchestra.run.smithi170.stdout: setuser = ceph 2023-10-12T18:45:57.682 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2023-10-12T18:45:57.682 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2023-10-12T18:45:57.682 INFO:teuthology.orchestra.run.smithi170.stdout: osd_max_omap_entries_per_request = 10 2023-10-12T18:45:57.683 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2023-10-12T18:45:57.683 INFO:teuthology.orchestra.run.smithi170.stdout: osd_max_omap_entries_per_request = 1000 2023-10-12T18:45:57.683 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2023-10-12T18:45:57.683 INFO:teuthology.orchestra.run.smithi170.stdout: osd_max_omap_entries_per_request = 10000 2023-10-12T18:45:57.694 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2023-10-12T18:45:57.694 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2023-10-12T18:45:57.694 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-12T18:45:57.695 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2023-10-12T18:45:57.695 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.695 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2023-10-12T18:45:57.695 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.695 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2023-10-12T18:45:57.696 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.696 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2023-10-12T18:45:57.696 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2023-10-12T18:45:57.696 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.696 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2023-10-12T18:45:57.697 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2023-10-12T18:45:57.697 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.697 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-12T18:45:57.697 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2023-10-12T18:45:57.697 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.698 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2023-10-12T18:45:57.698 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-12T18:45:57.698 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.698 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2023-10-12T18:45:57.698 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.699 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2023-10-12T18:45:57.699 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2023-10-12T18:45:57.699 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2023-10-12T18:45:57.699 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2023-10-12T18:45:57.699 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2023-10-12T18:45:57.699 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2023-10-12T18:45:57.700 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2023-10-12T18:45:57.700 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-12T18:45:57.700 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-12T18:45:57.700 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2023-10-12T18:45:57.700 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.701 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-10-12T18:45:57.701 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.701 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2023-10-12T18:45:57.701 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.701 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2023-10-12T18:45:57.702 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.702 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2023-10-12T18:45:57.702 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2023-10-12T18:45:57.702 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2023-10-12T18:45:57.702 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = 172.21.15.187 2023-10-12T18:45:57.703 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 171d6df1-cef5-40ec-977b-bbc477f9da52 2023-10-12T18:45:57.703 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.703 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2023-10-12T18:45:57.703 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2023-10-12T18:45:57.703 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.704 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2023-10-12T18:45:57.704 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2023-10-12T18:45:57.704 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.704 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2023-10-12T18:45:57.704 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2023-10-12T18:45:57.704 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.705 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2023-10-12T18:45:57.705 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2023-10-12T18:45:57.705 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2023-10-12T18:45:57.705 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.705 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2023-10-12T18:45:57.706 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2023-10-12T18:45:57.706 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.706 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2023-10-12T18:45:57.706 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.706 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2023-10-12T18:45:57.707 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.707 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2023-10-12T18:45:57.707 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.707 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2023-10-12T18:45:57.707 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2023-10-12T18:45:57.708 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.708 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2023-10-12T18:45:57.708 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2023-10-12T18:45:57.708 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2023-10-12T18:45:57.708 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2023-10-12T18:45:57.708 INFO:teuthology.orchestra.run.smithi187.stdout: rocksdb delete range threshold = 0 2023-10-12T18:45:57.709 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.709 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2023-10-12T18:45:57.709 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2023-10-12T18:45:57.709 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2023-10-12T18:45:57.709 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2023-10-12T18:45:57.710 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2023-10-12T18:45:57.710 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2023-10-12T18:45:57.710 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2023-10-12T18:45:57.710 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2023-10-12T18:45:57.710 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.711 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2023-10-12T18:45:57.711 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2023-10-12T18:45:57.711 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2023-10-12T18:45:57.711 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2023-10-12T18:45:57.711 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2023-10-12T18:45:57.712 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2023-10-12T18:45:57.712 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2023-10-12T18:45:57.712 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2023-10-12T18:45:57.712 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2023-10-12T18:45:57.712 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2023-10-12T18:45:57.712 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2023-10-12T18:45:57.713 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.713 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-12T18:45:57.713 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2023-10-12T18:45:57.713 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2023-10-12T18:45:57.713 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.714 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2023-10-12T18:45:57.714 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-12T18:45:57.714 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-12T18:45:57.714 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:45:57.714 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2023-10-12T18:45:57.715 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2023-10-12T18:45:57.715 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2023-10-12T18:45:57.715 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2023-10-12T18:45:57.715 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-12T18:45:57.715 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-12T18:45:57.715 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw = 20 2023-10-12T18:45:57.716 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt require ssl = False 2023-10-12T18:45:57.716 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt s3 kms backend = testing 2023-10-12T18:45:57.716 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T18:45:57.716 INFO:teuthology.orchestra.run.smithi187.stdout: rgw curl low speed time = 300 2023-10-12T18:45:57.716 INFO:teuthology.orchestra.run.smithi187.stdout: rgw data log num shards = 4 2023-10-12T18:45:57.717 INFO:teuthology.orchestra.run.smithi187.stdout: rgw md log max shards = 4 2023-10-12T18:45:57.717 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync log trim interval = 0 2023-10-12T18:45:57.717 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync meta inject err probability = 0.1 2023-10-12T18:45:57.717 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync obj etag verify = True 2023-10-12T18:45:57.717 INFO:teuthology.orchestra.run.smithi187.stdout: setgroup = ceph 2023-10-12T18:45:57.718 INFO:teuthology.orchestra.run.smithi187.stdout: setuser = ceph 2023-10-12T18:45:57.718 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2023-10-12T18:45:57.718 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2023-10-12T18:45:57.718 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 10 2023-10-12T18:45:57.718 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2023-10-12T18:45:57.719 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 1000 2023-10-12T18:45:57.719 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2023-10-12T18:45:57.719 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 10000 2023-10-12T18:45:57.720 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-10-12T18:45:57.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-10-12T18:45:57.810 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-12T18:45:57.810 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:57.810 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-10-12T18:45:57.825 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:57.825 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-10-12T18:45:57.879 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T18:45:57.879 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:57.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/c2.keyring 2023-10-12T18:45:57.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-12T18:45:57.915 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-12T18:45:57.915 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-12T18:45:57.971 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T18:45:57.972 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:57.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/c2.keyring 2023-10-12T18:45:57.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-12T18:45:58.006 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:45:58.006 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-12T18:45:58.060 INFO:tasks.ceph:Setting up mon nodes... 2023-10-12T18:45:58.061 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-12T18:45:58.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-10-12T18:45:58.147 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-10-12T18:45:58.150 INFO:tasks.ceph:Setting up mds nodes... 2023-10-12T18:45:58.150 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-12T18:45:58.150 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-10-12T18:45:58.187 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/c2.client.0.keyring 2023-10-12T18:45:58.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-10-12T18:45:58.277 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/c2.client.1.keyring 2023-10-12T18:45:58.344 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-12T18:45:58.344 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-12T18:45:58.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-10-12T18:45:58.408 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T18:45:58.408 INFO:tasks.ceph:role: c2.osd.0 2023-10-12T18:45:58.409 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T18:45:58.409 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-12T18:45:58.479 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T18:45:58.480 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T18:45:58.480 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T18:45:58.480 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 2023-10-12T18:45:58.480 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T18:45:58.480 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2023-10-12T18:45:58.481 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T18:45:58.481 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T18:45:58.481 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T18:45:58.481 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T18:45:58.913 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2023-10-12T18:45:58.915 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2023-10-12T18:45:58.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-10-12T18:45:59.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-10-12T18:45:59.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-10-12T18:45:59.153 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T18:45:59.154 INFO:tasks.ceph:role: c2.osd.1 2023-10-12T18:45:59.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T18:45:59.155 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-12T18:45:59.226 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T18:45:59.226 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T18:45:59.227 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T18:45:59.227 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 2023-10-12T18:45:59.227 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T18:45:59.227 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2023-10-12T18:45:59.227 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T18:45:59.227 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T18:45:59.228 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T18:45:59.228 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T18:45:59.647 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2023-10-12T18:45:59.648 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2023-10-12T18:45:59.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-10-12T18:45:59.731 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-10-12T18:45:59.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-10-12T18:45:59.864 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T18:45:59.864 INFO:tasks.ceph:role: c2.osd.2 2023-10-12T18:45:59.865 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T18:45:59.865 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-12T18:45:59.935 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T18:45:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T18:45:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T18:45:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 2023-10-12T18:45:59.936 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T18:45:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2023-10-12T18:45:59.937 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T18:45:59.937 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T18:45:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T18:45:59.937 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T18:46:00.379 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2023-10-12T18:46:00.381 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2023-10-12T18:46:00.382 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-10-12T18:46:00.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-10-12T18:46:00.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-12T18:46:00.620 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:00.620+0000 7fc0fd7cc200 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-10-12T18:46:00.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:00.622+0000 7fc0fd7cc200 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-10-12T18:46:00.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:00.622+0000 7fc0fd7cc200 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-12T18:46:00.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:00.622+0000 7fc0fd7cc200 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-12T18:46:00.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:00.622+0000 7fc0fd7cc200 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-12T18:46:00.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:00.623+0000 7fc0fd7cc200 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-10-12T18:46:02.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-10-12T18:46:02.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-12T18:46:02.806 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:02.804+0000 7f29796ee200 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-10-12T18:46:02.806 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:02.804+0000 7f29796ee200 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-10-12T18:46:02.806 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:02.804+0000 7f29796ee200 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-12T18:46:02.806 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:02.804+0000 7f29796ee200 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-12T18:46:02.807 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:02.804+0000 7f29796ee200 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-12T18:46:02.807 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:02.805+0000 7f29796ee200 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-10-12T18:46:04.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-10-12T18:46:04.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-12T18:46:04.991 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:04.989+0000 7f0f961fb200 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-10-12T18:46:04.991 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:04.989+0000 7f0f961fb200 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-10-12T18:46:04.991 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:04.989+0000 7f0f961fb200 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-12T18:46:04.992 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:04.989+0000 7f0f961fb200 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-12T18:46:04.992 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:04.989+0000 7f0f961fb200 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-12T18:46:04.992 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:04.990+0000 7f0f961fb200 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-10-12T18:46:07.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-10-12T18:46:07.084 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-12T18:46:07.085 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:46:07.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-10-12T18:46:07.150 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:46:07.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-10-12T18:46:07.214 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:46:07.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-10-12T18:46:07.278 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:46:07.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-10-12T18:46:07.342 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:46:07.342 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-10-12T18:46:07.396 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:46:07.397 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-10-12T18:46:07.452 INFO:tasks.ceph:Adding keys to all mons... 2023-10-12T18:46:07.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo tee -a /etc/ceph/c2.keyring 2023-10-12T18:46:07.515 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.x] 2023-10-12T18:46:07.515 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDmPihlzBzOCBAAy+RWW7pBPXY7sYQZsz+7Sg== 2023-10-12T18:46:07.515 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2023-10-12T18:46:07.515 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDoPihlf/f5JBAAqmc0Si4Se5Ql9LH56jcpAg== 2023-10-12T18:46:07.515 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2023-10-12T18:46:07.516 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDqPihlYPACMBAAbCHwWGIrzfMHH3OLaS8wpg== 2023-10-12T18:46:07.516 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2023-10-12T18:46:07.516 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDsPihlL64MOxAA0Oj7PG0yVZw3qluXFx2wfQ== 2023-10-12T18:46:07.516 INFO:teuthology.orchestra.run.smithi187.stdout:[client.0] 2023-10-12T18:46:07.516 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDmPihlm6cvCxAAyuBAfI1mMSuz6OUqNT34BQ== 2023-10-12T18:46:07.517 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2023-10-12T18:46:07.517 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDmPihlkeaCEBAAQDR1aOiqivi0LZLDOlP0CQ== 2023-10-12T18:46:07.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-12T18:46:07.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T18:46:07.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T18:46:07.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T18:46:07.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-12T18:46:07.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-12T18:46:07.840 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-12T18:46:07.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-10-12T18:46:07.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-10-12T18:46:07.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-10-12T18:46:07.979 DEBUG:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-10-12T18:46:08.034 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-10-12T18:46:08.035 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-10-12T18:46:08.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-10-12T18:46:08.076 INFO:tasks.ceph.c2.mon.a:Started 2023-10-12T18:46:08.077 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-10-12T18:46:08.077 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-10-12T18:46:08.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-10-12T18:46:08.079 INFO:tasks.ceph.c2.mgr.x:Started 2023-10-12T18:46:08.080 INFO:tasks.ceph:Setting crush tunables to default 2023-10-12T18:46:08.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd crush tunables default 2023-10-12T18:46:08.285 INFO:teuthology.orchestra.run.smithi187.stderr:adjusted tunables profile to default 2023-10-12T18:46:08.295 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-10-12T18:46:08.296 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:46:08.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-10-12T18:46:08.320 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:46:08.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-10-12T18:46:08.385 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-12T18:46:08.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-10-12T18:46:08.450 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd new 94cdd9ee-c5f5-4302-8aa5-05e5993dd79a 0 2023-10-12T18:46:08.491 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:08.490+0000 7fe013bdd680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-12T18:46:08.564 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:08.563+0000 7fe013bdd680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-12T18:46:08.693 INFO:teuthology.orchestra.run.smithi187.stdout:0 2023-10-12T18:46:08.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd new 468de964-ef0f-4312-8eb7-8fe606aa4125 1 2023-10-12T18:46:08.907 INFO:teuthology.orchestra.run.smithi187.stdout:1 2023-10-12T18:46:08.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd new 184e7d87-7a36-412d-8bee-6e3391d5909c 2 2023-10-12T18:46:09.061 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:09.058+0000 7fe013bdd680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-12T18:46:09.121 INFO:teuthology.orchestra.run.smithi187.stdout:2 2023-10-12T18:46:09.133 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-10-12T18:46:09.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-10-12T18:46:09.136 INFO:tasks.ceph.c2.osd.0:Started 2023-10-12T18:46:09.136 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-10-12T18:46:09.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-10-12T18:46:09.139 INFO:tasks.ceph.c2.osd.1:Started 2023-10-12T18:46:09.140 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-10-12T18:46:09.140 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-10-12T18:46:09.142 INFO:tasks.ceph.c2.osd.2:Started 2023-10-12T18:46:09.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-12T18:46:09.339 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:46:09.340 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":5,"fsid":"171d6df1-cef5-40ec-977b-bbc477f9da52","created":"2023-10-12T18:46:08.198710+0000","modified":"2023-10-12T18:46:09.118839+0000","last_up_change":"0.000000","last_in_change":"2023-10-12T18:46:09.118839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"94cdd9ee-c5f5-4302-8aa5-05e5993dd79a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"468de964-ef0f-4312-8eb7-8fe606aa4125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"184e7d87-7a36-412d-8bee-6e3391d5909c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:46:09.352 INFO:tasks.ceph.ceph_manager.c2:[] 2023-10-12T18:46:09.352 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-12T18:46:09.655 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-12T18:46:09.730 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2023-10-12T18:46:09.728+0000 7f8e57a5c200 -1 Falling back to public interface 2023-10-12T18:46:09.757 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2023-10-12T18:46:09.756+0000 7f5dcd084200 -1 Falling back to public interface 2023-10-12T18:46:09.759 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2023-10-12T18:46:09.758+0000 7f59ac73d200 -1 Falling back to public interface 2023-10-12T18:46:09.840 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:09.839+0000 7fe013bdd680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-12T18:46:09.867 INFO:teuthology.misc.health.smithi187.stdout: 2023-10-12T18:46:09.867 INFO:teuthology.misc.health.smithi187.stdout:{"epoch":5,"fsid":"171d6df1-cef5-40ec-977b-bbc477f9da52","created":"2023-10-12T18:46:08.198710+0000","modified":"2023-10-12T18:46:09.118839+0000","last_up_change":"0.000000","last_in_change":"2023-10-12T18:46:09.118839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"94cdd9ee-c5f5-4302-8aa5-05e5993dd79a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"468de964-ef0f-4312-8eb7-8fe606aa4125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"184e7d87-7a36-412d-8bee-6e3391d5909c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:46:09.878 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-12T18:46:09.909 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:09.908+0000 7fe013bdd680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-12T18:46:10.099 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:10.098+0000 7fe013bdd680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-12T18:46:10.550 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:10.549+0000 7fe013bdd680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-12T18:46:10.653 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:10.652+0000 7fe013bdd680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-12T18:46:10.807 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:10.805+0000 7fe013bdd680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-12T18:46:10.885 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:10.883+0000 7fe013bdd680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-12T18:46:10.954 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:10.952+0000 7fe013bdd680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-12T18:46:11.218 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:11.216+0000 7fe013bdd680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-12T18:46:11.421 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:11.420+0000 7fe013bdd680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-12T18:46:12.087 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:12.086+0000 7fe013bdd680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-12T18:46:12.157 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:12.155+0000 7fe013bdd680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-12T18:46:12.247 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:12.245+0000 7fe013bdd680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-12T18:46:12.395 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:12.393+0000 7fe013bdd680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-12T18:46:12.438 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2023-10-12T18:46:12.437+0000 7f8e57a5c200 -1 osd.0 0 log_to_monitors {default=true} 2023-10-12T18:46:12.467 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:12.466+0000 7fe013bdd680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-12T18:46:12.478 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2023-10-12T18:46:12.477+0000 7f5dcd084200 -1 osd.1 0 log_to_monitors {default=true} 2023-10-12T18:46:12.506 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2023-10-12T18:46:12.505+0000 7f59ac73d200 -1 osd.2 0 log_to_monitors {default=true} 2023-10-12T18:46:12.589 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:12.588+0000 7fe013bdd680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-12T18:46:12.696 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:12.695+0000 7fe013bdd680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-12T18:46:13.016 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:13.015+0000 7fe013bdd680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-12T18:46:13.087 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:13.086+0000 7fe013bdd680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-12T18:46:13.495 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2023-10-12T18:46:13.494+0000 7f8e4e419700 -1 osd.0 0 waiting for initial osdmap 2023-10-12T18:46:13.496 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2023-10-12T18:46:13.494+0000 7f5dc5244700 -1 osd.1 0 waiting for initial osdmap 2023-10-12T18:46:13.498 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2023-10-12T18:46:13.497+0000 7f8e4ab95700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T18:46:13.501 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2023-10-12T18:46:13.500+0000 7f5dc01bd700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T18:46:14.105 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2023-10-12T18:46:14.104+0000 7fdfc9e22700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-12T18:46:14.497 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2023-10-12T18:46:14.496+0000 7f59a30fa700 -1 osd.2 0 waiting for initial osdmap 2023-10-12T18:46:14.531 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2023-10-12T18:46:14.530+0000 7f599f876700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T18:46:16.182 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-12T18:46:16.468 INFO:teuthology.misc.health.smithi187.stdout: 2023-10-12T18:46:16.468 INFO:teuthology.misc.health.smithi187.stdout:{"epoch":9,"fsid":"171d6df1-cef5-40ec-977b-bbc477f9da52","created":"2023-10-12T18:46:08.198710+0000","modified":"2023-10-12T18:46:15.493577+0000","last_up_change":"2023-10-12T18:46:15.493577+0000","last_in_change":"2023-10-12T18:46:09.118839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-12T18:46:14.529244+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"94cdd9ee-c5f5-4302-8aa5-05e5993dd79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":86277},{"type":"v1","addr":"172.21.15.187:6801","nonce":86277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":86277},{"type":"v1","addr":"172.21.15.187:6803","nonce":86277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":86277},{"type":"v1","addr":"172.21.15.187:6807","nonce":86277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":86277},{"type":"v1","addr":"172.21.15.187:6805","nonce":86277}]},"public_addr":"172.21.15.187:6801/86277","cluster_addr":"172.21.15.187:6803/86277","heartbeat_back_addr":"172.21.15.187:6807/86277","heartbeat_front_addr":"172.21.15.187:6805/86277","state":["exists","up"]},{"osd":1,"uuid":"468de964-ef0f-4312-8eb7-8fe606aa4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":86292},{"type":"v1","addr":"172.21.15.187:6809","nonce":86292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":86292},{"type":"v1","addr":"172.21.15.187:6811","nonce":86292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":86292},{"type":"v1","addr":"172.21.15.187:6815","nonce":86292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":86292},{"type":"v1","addr":"172.21.15.187:6813","nonce":86292}]},"public_addr":"172.21.15.187:6809/86292","cluster_addr":"172.21.15.187:6811/86292","heartbeat_back_addr":"172.21.15.187:6815/86292","heartbeat_front_addr":"172.21.15.187:6813/86292","state":["exists","up"]},{"osd":2,"uuid":"184e7d87-7a36-412d-8bee-6e3391d5909c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":86293},{"type":"v1","addr":"172.21.15.187:6817","nonce":86293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":86293},{"type":"v1","addr":"172.21.15.187:6819","nonce":86293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":86293},{"type":"v1","addr":"172.21.15.187:6823","nonce":86293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":86293},{"type":"v1","addr":"172.21.15.187:6821","nonce":86293}]},"public_addr":"172.21.15.187:6817/86293","cluster_addr":"172.21.15.187:6819/86293","heartbeat_back_addr":"172.21.15.187:6823/86293","heartbeat_front_addr":"172.21.15.187:6821/86293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.465248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.456915+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:46:16.480 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-12T18:46:16.481 INFO:tasks.ceph:Creating RBD pool 2023-10-12T18:46:16.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd pool create rbd 8 2023-10-12T18:46:17.511 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'rbd' created 2023-10-12T18:46:17.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-12T18:46:18.513 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-12T18:46:18.526 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-10-12T18:46:18.527 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-10-12T18:46:18.527 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-10-12T18:46:18.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-10-12T18:46:18.529 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-12T18:46:18.868 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:46:18.880 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":85940},{"type":"v1","addr":"172.21.15.187:6825","nonce":85940}]},"active_addr":"172.21.15.187:6825/85940","active_change":"2023-10-12T18:46:13.099869+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":984134018}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":4287839205}]}]} 2023-10-12T18:46:18.884 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-10-12T18:46:18.885 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-10-12T18:46:18.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-12T18:46:19.170 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:46:19.171 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":12,"fsid":"171d6df1-cef5-40ec-977b-bbc477f9da52","created":"2023-10-12T18:46:08.198710+0000","modified":"2023-10-12T18:46:18.502628+0000","last_up_change":"2023-10-12T18:46:15.493577+0000","last_in_change":"2023-10-12T18:46:09.118839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-12T18:46:14.529244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T18:46:16.798729+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"94cdd9ee-c5f5-4302-8aa5-05e5993dd79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":86277},{"type":"v1","addr":"172.21.15.187:6801","nonce":86277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":86277},{"type":"v1","addr":"172.21.15.187:6803","nonce":86277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":86277},{"type":"v1","addr":"172.21.15.187:6807","nonce":86277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":86277},{"type":"v1","addr":"172.21.15.187:6805","nonce":86277}]},"public_addr":"172.21.15.187:6801/86277","cluster_addr":"172.21.15.187:6803/86277","heartbeat_back_addr":"172.21.15.187:6807/86277","heartbeat_front_addr":"172.21.15.187:6805/86277","state":["exists","up"]},{"osd":1,"uuid":"468de964-ef0f-4312-8eb7-8fe606aa4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":86292},{"type":"v1","addr":"172.21.15.187:6809","nonce":86292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":86292},{"type":"v1","addr":"172.21.15.187:6811","nonce":86292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":86292},{"type":"v1","addr":"172.21.15.187:6815","nonce":86292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":86292},{"type":"v1","addr":"172.21.15.187:6813","nonce":86292}]},"public_addr":"172.21.15.187:6809/86292","cluster_addr":"172.21.15.187:6811/86292","heartbeat_back_addr":"172.21.15.187:6815/86292","heartbeat_front_addr":"172.21.15.187:6813/86292","state":["exists","up"]},{"osd":2,"uuid":"184e7d87-7a36-412d-8bee-6e3391d5909c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":86293},{"type":"v1","addr":"172.21.15.187:6817","nonce":86293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":86293},{"type":"v1","addr":"172.21.15.187:6819","nonce":86293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":86293},{"type":"v1","addr":"172.21.15.187:6823","nonce":86293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":86293},{"type":"v1","addr":"172.21.15.187:6821","nonce":86293}]},"public_addr":"172.21.15.187:6817/86293","cluster_addr":"172.21.15.187:6819/86293","heartbeat_back_addr":"172.21.15.187:6823/86293","heartbeat_front_addr":"172.21.15.187:6821/86293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.465248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.456915+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.507348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:46:19.184 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-10-12T18:46:19.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-12T18:46:19.469 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:46:19.470 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":12,"fsid":"171d6df1-cef5-40ec-977b-bbc477f9da52","created":"2023-10-12T18:46:08.198710+0000","modified":"2023-10-12T18:46:18.502628+0000","last_up_change":"2023-10-12T18:46:15.493577+0000","last_in_change":"2023-10-12T18:46:09.118839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-12T18:46:14.529244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T18:46:16.798729+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"94cdd9ee-c5f5-4302-8aa5-05e5993dd79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":86277},{"type":"v1","addr":"172.21.15.187:6801","nonce":86277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":86277},{"type":"v1","addr":"172.21.15.187:6803","nonce":86277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":86277},{"type":"v1","addr":"172.21.15.187:6807","nonce":86277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":86277},{"type":"v1","addr":"172.21.15.187:6805","nonce":86277}]},"public_addr":"172.21.15.187:6801/86277","cluster_addr":"172.21.15.187:6803/86277","heartbeat_back_addr":"172.21.15.187:6807/86277","heartbeat_front_addr":"172.21.15.187:6805/86277","state":["exists","up"]},{"osd":1,"uuid":"468de964-ef0f-4312-8eb7-8fe606aa4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":86292},{"type":"v1","addr":"172.21.15.187:6809","nonce":86292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":86292},{"type":"v1","addr":"172.21.15.187:6811","nonce":86292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":86292},{"type":"v1","addr":"172.21.15.187:6815","nonce":86292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":86292},{"type":"v1","addr":"172.21.15.187:6813","nonce":86292}]},"public_addr":"172.21.15.187:6809/86292","cluster_addr":"172.21.15.187:6811/86292","heartbeat_back_addr":"172.21.15.187:6815/86292","heartbeat_front_addr":"172.21.15.187:6813/86292","state":["exists","up"]},{"osd":2,"uuid":"184e7d87-7a36-412d-8bee-6e3391d5909c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":86293},{"type":"v1","addr":"172.21.15.187:6817","nonce":86293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":86293},{"type":"v1","addr":"172.21.15.187:6819","nonce":86293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":86293},{"type":"v1","addr":"172.21.15.187:6823","nonce":86293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":86293},{"type":"v1","addr":"172.21.15.187:6821","nonce":86293}]},"public_addr":"172.21.15.187:6817/86293","cluster_addr":"172.21.15.187:6819/86293","heartbeat_back_addr":"172.21.15.187:6823/86293","heartbeat_front_addr":"172.21.15.187:6821/86293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.465248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.456915+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.507348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T18:46:19.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-10-12T18:46:19.630 INFO:teuthology.orchestra.run.smithi187.stdout:34359738371 2023-10-12T18:46:19.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-10-12T18:46:19.775 INFO:teuthology.orchestra.run.smithi187.stdout:34359738371 2023-10-12T18:46:19.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-10-12T18:46:19.919 INFO:teuthology.orchestra.run.smithi187.stdout:38654705667 2023-10-12T18:46:19.920 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-12T18:46:20.204 INFO:teuthology.orchestra.run.smithi187.stdout:34359738370 2023-10-12T18:46:20.218 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2023-10-12T18:46:21.219 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-12T18:46:21.506 INFO:teuthology.orchestra.run.smithi187.stdout:34359738370 2023-10-12T18:46:21.520 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2023-10-12T18:46:22.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-12T18:46:22.813 INFO:teuthology.orchestra.run.smithi187.stdout:34359738372 2023-10-12T18:46:22.826 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738372 for osd.0 2023-10-12T18:46:22.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-10-12T18:46:23.110 INFO:teuthology.orchestra.run.smithi187.stdout:34359738372 2023-10-12T18:46:23.123 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738372 for osd.1 2023-10-12T18:46:23.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-10-12T18:46:23.410 INFO:teuthology.orchestra.run.smithi187.stdout:38654705668 2023-10-12T18:46:23.424 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705668 for osd.2 2023-10-12T18:46:23.424 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-10-12T18:46:23.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-12T18:46:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:46:23.710 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2023-10-12T18:46:23.720 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-10-12T18:46:23.104766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681200,"statfs":{"total":322122547200,"available":321209548800,"internally_reserved":0,"allocated":319488,"data_stored":87756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.518404+0000","last_change":"2023-10-12T18:46:18.510954+0000","last_active":"2023-10-12T18:46:18.518404+0000","last_peered":"2023-10-12T18:46:18.518404+0000","last_clean":"2023-10-12T18:46:18.518404+0000","last_became_active":"2023-10-12T18:46:18.510668+0000","last_became_peered":"2023-10-12T18:46:18.510668+0000","last_unstale":"2023-10-12T18:46:18.518404+0000","last_undegraded":"2023-10-12T18:46:18.518404+0000","last_fullsized":"2023-10-12T18:46:18.518404+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.510470+0000","last_change":"2023-10-12T18:46:18.508633+0000","last_active":"2023-10-12T18:46:18.510470+0000","last_peered":"2023-10-12T18:46:18.510470+0000","last_clean":"2023-10-12T18:46:18.510470+0000","last_became_active":"2023-10-12T18:46:18.508332+0000","last_became_peered":"2023-10-12T18:46:18.508332+0000","last_unstale":"2023-10-12T18:46:18.510470+0000","last_undegraded":"2023-10-12T18:46:18.510470+0000","last_fullsized":"2023-10-12T18:46:18.510470+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.518616+0000","last_change":"2023-10-12T18:46:18.511990+0000","last_active":"2023-10-12T18:46:18.518616+0000","last_peered":"2023-10-12T18:46:18.518616+0000","last_clean":"2023-10-12T18:46:18.518616+0000","last_became_active":"2023-10-12T18:46:18.511719+0000","last_became_peered":"2023-10-12T18:46:18.511719+0000","last_unstale":"2023-10-12T18:46:18.518616+0000","last_undegraded":"2023-10-12T18:46:18.518616+0000","last_fullsized":"2023-10-12T18:46:18.518616+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.508377+0000","last_change":"2023-10-12T18:46:18.508377+0000","last_active":"2023-10-12T18:46:18.508377+0000","last_peered":"2023-10-12T18:46:18.508377+0000","last_clean":"2023-10-12T18:46:18.508377+0000","last_became_active":"2023-10-12T18:46:18.508116+0000","last_became_peered":"2023-10-12T18:46:18.508116+0000","last_unstale":"2023-10-12T18:46:18.508377+0000","last_undegraded":"2023-10-12T18:46:18.508377+0000","last_fullsized":"2023-10-12T18:46:18.508377+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.506774+0000","last_change":"2023-10-12T18:46:16.500738+0000","last_active":"2023-10-12T18:46:18.506774+0000","last_peered":"2023-10-12T18:46:18.506774+0000","last_clean":"2023-10-12T18:46:18.506774+0000","last_became_active":"2023-10-12T18:46:16.500431+0000","last_became_peered":"2023-10-12T18:46:16.500431+0000","last_unstale":"2023-10-12T18:46:18.506774+0000","last_undegraded":"2023-10-12T18:46:18.506774+0000","last_fullsized":"2023-10-12T18:46:18.506774+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:15.493577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:15.493577+0000","last_clean_scrub_stamp":"2023-10-12T18:46:15.493577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.518408+0000","last_change":"2023-10-12T18:46:18.510942+0000","last_active":"2023-10-12T18:46:18.518408+0000","last_peered":"2023-10-12T18:46:18.518408+0000","last_clean":"2023-10-12T18:46:18.518408+0000","last_became_active":"2023-10-12T18:46:18.510658+0000","last_became_peered":"2023-10-12T18:46:18.510658+0000","last_unstale":"2023-10-12T18:46:18.518408+0000","last_undegraded":"2023-10-12T18:46:18.518408+0000","last_fullsized":"2023-10-12T18:46:18.518408+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.510135+0000","last_change":"2023-10-12T18:46:18.510135+0000","last_active":"2023-10-12T18:46:18.510135+0000","last_peered":"2023-10-12T18:46:18.510135+0000","last_clean":"2023-10-12T18:46:18.510135+0000","last_became_active":"2023-10-12T18:46:18.509785+0000","last_became_peered":"2023-10-12T18:46:18.509785+0000","last_unstale":"2023-10-12T18:46:18.510135+0000","last_undegraded":"2023-10-12T18:46:18.510135+0000","last_fullsized":"2023-10-12T18:46:18.510135+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.511955+0000","last_change":"2023-10-12T18:46:18.511955+0000","last_active":"2023-10-12T18:46:18.511955+0000","last_peered":"2023-10-12T18:46:18.511955+0000","last_clean":"2023-10-12T18:46:18.511955+0000","last_became_active":"2023-10-12T18:46:18.511697+0000","last_became_peered":"2023-10-12T18:46:18.511697+0000","last_unstale":"2023-10-12T18:46:18.511955+0000","last_undegraded":"2023-10-12T18:46:18.511955+0000","last_fullsized":"2023-10-12T18:46:18.511955+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.518411+0000","last_change":"2023-10-12T18:46:18.511036+0000","last_active":"2023-10-12T18:46:18.518411+0000","last_peered":"2023-10-12T18:46:18.518411+0000","last_clean":"2023-10-12T18:46:18.518411+0000","last_became_active":"2023-10-12T18:46:18.510740+0000","last_became_peered":"2023-10-12T18:46:18.510740+0000","last_unstale":"2023-10-12T18:46:18.518411+0000","last_undegraded":"2023-10-12T18:46:18.518411+0000","last_fullsized":"2023-10-12T18:46:18.518411+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-12T18:46:23.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-12T18:46:24.003 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:46:24.005 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2023-10-12T18:46:24.016 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-10-12T18:46:23.104766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681200,"statfs":{"total":322122547200,"available":321209548800,"internally_reserved":0,"allocated":319488,"data_stored":87756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.518404+0000","last_change":"2023-10-12T18:46:18.510954+0000","last_active":"2023-10-12T18:46:18.518404+0000","last_peered":"2023-10-12T18:46:18.518404+0000","last_clean":"2023-10-12T18:46:18.518404+0000","last_became_active":"2023-10-12T18:46:18.510668+0000","last_became_peered":"2023-10-12T18:46:18.510668+0000","last_unstale":"2023-10-12T18:46:18.518404+0000","last_undegraded":"2023-10-12T18:46:18.518404+0000","last_fullsized":"2023-10-12T18:46:18.518404+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.510470+0000","last_change":"2023-10-12T18:46:18.508633+0000","last_active":"2023-10-12T18:46:18.510470+0000","last_peered":"2023-10-12T18:46:18.510470+0000","last_clean":"2023-10-12T18:46:18.510470+0000","last_became_active":"2023-10-12T18:46:18.508332+0000","last_became_peered":"2023-10-12T18:46:18.508332+0000","last_unstale":"2023-10-12T18:46:18.510470+0000","last_undegraded":"2023-10-12T18:46:18.510470+0000","last_fullsized":"2023-10-12T18:46:18.510470+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.518616+0000","last_change":"2023-10-12T18:46:18.511990+0000","last_active":"2023-10-12T18:46:18.518616+0000","last_peered":"2023-10-12T18:46:18.518616+0000","last_clean":"2023-10-12T18:46:18.518616+0000","last_became_active":"2023-10-12T18:46:18.511719+0000","last_became_peered":"2023-10-12T18:46:18.511719+0000","last_unstale":"2023-10-12T18:46:18.518616+0000","last_undegraded":"2023-10-12T18:46:18.518616+0000","last_fullsized":"2023-10-12T18:46:18.518616+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.508377+0000","last_change":"2023-10-12T18:46:18.508377+0000","last_active":"2023-10-12T18:46:18.508377+0000","last_peered":"2023-10-12T18:46:18.508377+0000","last_clean":"2023-10-12T18:46:18.508377+0000","last_became_active":"2023-10-12T18:46:18.508116+0000","last_became_peered":"2023-10-12T18:46:18.508116+0000","last_unstale":"2023-10-12T18:46:18.508377+0000","last_undegraded":"2023-10-12T18:46:18.508377+0000","last_fullsized":"2023-10-12T18:46:18.508377+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.506774+0000","last_change":"2023-10-12T18:46:16.500738+0000","last_active":"2023-10-12T18:46:18.506774+0000","last_peered":"2023-10-12T18:46:18.506774+0000","last_clean":"2023-10-12T18:46:18.506774+0000","last_became_active":"2023-10-12T18:46:16.500431+0000","last_became_peered":"2023-10-12T18:46:16.500431+0000","last_unstale":"2023-10-12T18:46:18.506774+0000","last_undegraded":"2023-10-12T18:46:18.506774+0000","last_fullsized":"2023-10-12T18:46:18.506774+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:15.493577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:15.493577+0000","last_clean_scrub_stamp":"2023-10-12T18:46:15.493577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.518408+0000","last_change":"2023-10-12T18:46:18.510942+0000","last_active":"2023-10-12T18:46:18.518408+0000","last_peered":"2023-10-12T18:46:18.518408+0000","last_clean":"2023-10-12T18:46:18.518408+0000","last_became_active":"2023-10-12T18:46:18.510658+0000","last_became_peered":"2023-10-12T18:46:18.510658+0000","last_unstale":"2023-10-12T18:46:18.518408+0000","last_undegraded":"2023-10-12T18:46:18.518408+0000","last_fullsized":"2023-10-12T18:46:18.518408+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.510135+0000","last_change":"2023-10-12T18:46:18.510135+0000","last_active":"2023-10-12T18:46:18.510135+0000","last_peered":"2023-10-12T18:46:18.510135+0000","last_clean":"2023-10-12T18:46:18.510135+0000","last_became_active":"2023-10-12T18:46:18.509785+0000","last_became_peered":"2023-10-12T18:46:18.509785+0000","last_unstale":"2023-10-12T18:46:18.510135+0000","last_undegraded":"2023-10-12T18:46:18.510135+0000","last_fullsized":"2023-10-12T18:46:18.510135+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.511955+0000","last_change":"2023-10-12T18:46:18.511955+0000","last_active":"2023-10-12T18:46:18.511955+0000","last_peered":"2023-10-12T18:46:18.511955+0000","last_clean":"2023-10-12T18:46:18.511955+0000","last_became_active":"2023-10-12T18:46:18.511697+0000","last_became_peered":"2023-10-12T18:46:18.511697+0000","last_unstale":"2023-10-12T18:46:18.511955+0000","last_undegraded":"2023-10-12T18:46:18.511955+0000","last_fullsized":"2023-10-12T18:46:18.511955+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T18:46:18.518411+0000","last_change":"2023-10-12T18:46:18.511036+0000","last_active":"2023-10-12T18:46:18.518411+0000","last_peered":"2023-10-12T18:46:18.518411+0000","last_clean":"2023-10-12T18:46:18.518411+0000","last_became_active":"2023-10-12T18:46:18.510740+0000","last_became_peered":"2023-10-12T18:46:18.510740+0000","last_unstale":"2023-10-12T18:46:18.518411+0000","last_undegraded":"2023-10-12T18:46:18.518411+0000","last_fullsized":"2023-10-12T18:46:18.518411+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T18:46:17.495752+0000","last_clean_scrub_stamp":"2023-10-12T18:46:17.495752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-12T18:46:24.017 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-10-12T18:46:24.017 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-10-12T18:46:24.018 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-10-12T18:46:24.018 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-10-12T18:46:24.334 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T18:46:24.348 INFO:teuthology.orchestra.run.smithi187.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-12T18:46:24.348 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-10-12T18:46:24.349 INFO:teuthology.run_tasks:Running task rgw... 2023-10-12T18:46:24.362 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-10-12T18:46:24.363 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-10-12T18:46:24.363 INFO:tasks.rgw:Creating data pools 2023-10-12T18:46:24.364 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-10-12T18:46:24.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-12T18:46:25.089 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'default.rgw.buckets.data' created 2023-10-12T18:46:25.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-12T18:46:26.105 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-12T18:46:26.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-12T18:46:27.097 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'default.rgw.buckets.index' created 2023-10-12T18:46:27.108 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-12T18:46:28.107 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-12T18:46:28.119 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-10-12T18:46:28.121 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-12T18:46:28.409 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-12T18:46:28.419 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-12T18:46:29.090 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-12T18:46:29.101 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-12T18:46:29.388 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-12T18:46:29.399 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-12T18:46:30.092 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-12T18:46:30.103 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-10-12T18:46:30.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-12T18:46:31.173 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'default.rgw.buckets.data' created 2023-10-12T18:46:31.187 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-12T18:46:32.188 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-12T18:46:32.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-12T18:46:33.186 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'default.rgw.buckets.index' created 2023-10-12T18:46:33.198 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-12T18:46:34.200 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-12T18:46:34.214 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-10-12T18:46:34.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-12T18:46:34.505 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-12T18:46:34.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-12T18:46:35.186 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-12T18:46:35.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-12T18:46:35.488 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-12T18:46:35.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-12T18:46:36.187 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-12T18:46:36.198 DEBUG:tasks.rgw:Pools created 2023-10-12T18:46:36.199 INFO:tasks.rgw:Configuring compression type = random 2023-10-12T18:46:36.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-10-12T18:46:36.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-10-12T18:46:36.200 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-10-12T18:46:36.301 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:36.301 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:36.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:36.305+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.105+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.106+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.113+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.113+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.114+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.114+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.114+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 10 cannot find current period zonegroup using local zonegroup 2023-10-12T18:46:38.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.125+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.126+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.126+0000 7f2748261500 10 Creating default zonegroup 2023-10-12T18:46:38.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.126+0000 7f2748261500 20 get_system_obj_state: rctx=0x7ffd49662dd0 obj=:.pools.avail state=0x5636d51c4810 s->prefetch_data=0 2023-10-12T18:46:38.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.126+0000 7f2748261500 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-12T18:46:38.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.133+0000 7f2748261500 10 WARNING: store->list_zones() returned r=-2 2023-10-12T18:46:38.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.133+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.134+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.136+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.136+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.136+0000 7f2748261500 10 could not read realm id: (2) No such file or directory 2023-10-12T18:46:38.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.136+0000 7f2748261500 10 WARNING: failed to set zone as default, r=-22 2023-10-12T18:46:38.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.136+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.136+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 20 rados_obj.operate() r=0 bl.length=873 2023-10-12T18:46:38.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:38.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 20 zonegroup default 2023-10-12T18:46:38.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:38.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 10 Cannot find current period zone using local zone 2023-10-12T18:46:38.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 10 Using default name default 2023-10-12T18:46:38.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.139+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.140+0000 7f2748261500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:38.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.140+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:38.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.140+0000 7f2748261500 20 rados_obj.operate() r=0 bl.length=873 2023-10-12T18:46:38.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.140+0000 7f2748261500 20 zone default found 2023-10-12T18:46:38.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:38.140+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:40.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:40.127+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:40.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:40.127+0000 7f2748261500 20 rados->read ofs=0 len=0 2023-10-12T18:46:40.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:40.127+0000 7f2748261500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:40.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:40.127+0000 7f2748261500 20 started sync module instance, tier type = 2023-10-12T18:46:40.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:40.127+0000 7f2748261500 20 started zone id=ddc41caf-ed31-4401-8108-067c2eb0a43d (name=default) with tier type = 2023-10-12T18:46:42.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.140+0000 7f2748261500 20 add_watcher() i=0 2023-10-12T18:46:42.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.140+0000 7f2748261500 20 add_watcher() i=1 2023-10-12T18:46:42.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.140+0000 7f2748261500 20 add_watcher() i=2 2023-10-12T18:46:42.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.140+0000 7f2748261500 20 add_watcher() i=3 2023-10-12T18:46:42.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.140+0000 7f2748261500 20 add_watcher() i=4 2023-10-12T18:46:42.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.140+0000 7f2748261500 20 add_watcher() i=5 2023-10-12T18:46:42.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.140+0000 7f2748261500 20 add_watcher() i=6 2023-10-12T18:46:42.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.141+0000 7f2748261500 20 add_watcher() i=7 2023-10-12T18:46:42.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.141+0000 7f2748261500 2 all 8 watchers are set, enabling cache 2023-10-12T18:46:42.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.141+0000 7f2748261500 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):409: oid=data_loggenerations_metadata not found 2023-10-12T18:46:42.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.0 2023-10-12T18:46:42.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:42.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.0 does not exist 2023-10-12T18:46:42.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.1 2023-10-12T18:46:42.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:42.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.1 does not exist 2023-10-12T18:46:42.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.2 2023-10-12T18:46:42.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:42.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.2 does not exist 2023-10-12T18:46:42.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.3 2023-10-12T18:46:42.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:42.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.3 does not exist 2023-10-12T18:46:42.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.142+0000 7f2748261500 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1206 entering 2023-10-12T18:46:42.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.143+0000 7f2748261500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:42.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.146+0000 7f2704fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T18:46:42.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.146+0000 7f2748261500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T18:46:42.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:42.146+0000 7f2748261500 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T18:46:45.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.126+0000 7f2748261500 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-12T18:46:45.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.126+0000 7f2748261500 5 note: GC not initialized 2023-10-12T18:46:45.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.128+0000 7f26677fe700 20 reqs_thread_entry: start 2023-10-12T18:46:45.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.128+0000 7f26657fa700 20 reqs_thread_entry: start 2023-10-12T18:46:45.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.128+0000 7f2748261500 20 init_complete bucket index max shards: 11 2023-10-12T18:46:45.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.129+0000 7f2748261500 10 rgw notify: Started notification manager with: 1 workers 2023-10-12T18:46:45.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.129+0000 7f26577fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 12 18:47:15 2023 2023-10-12T18:46:45.131 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-12T18:46:45.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.129+0000 7f2656ffd700 20 reqs_thread_entry: start 2023-10-12T18:46:45.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.151+0000 7f2748261500 20 remove_watcher() i=0 2023-10-12T18:46:45.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.151+0000 7f2748261500 2 removed watcher, disabling cache 2023-10-12T18:46:45.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.152+0000 7f2748261500 20 remove_watcher() i=1 2023-10-12T18:46:45.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.153+0000 7f2748261500 20 remove_watcher() i=2 2023-10-12T18:46:45.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.154+0000 7f2748261500 20 remove_watcher() i=3 2023-10-12T18:46:45.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.155+0000 7f2748261500 20 remove_watcher() i=4 2023-10-12T18:46:45.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.156+0000 7f2748261500 20 remove_watcher() i=5 2023-10-12T18:46:45.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.157+0000 7f2748261500 20 remove_watcher() i=6 2023-10-12T18:46:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.158+0000 7f2748261500 20 remove_watcher() i=7 2023-10-12T18:46:45.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.159+0000 7f2704fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T18:46:45.164 INFO:teuthology.orchestra.run.smithi170.stdout:[] 2023-10-12T18:46:45.166 DEBUG:tasks.util.rgw: json result: [] 2023-10-12T18:46:45.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T18:46:45.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T18:46:45.167 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-12T18:46:45.212 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:45.213 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:45.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.217+0000 7ff2c4707500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T18:46:45.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.217+0000 7ff2c4707500 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T18:46:45.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.220+0000 7ff2117fa700 20 reqs_thread_entry: start 2023-10-12T18:46:45.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.220+0000 7ff2c4707500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.221+0000 7ff2c4707500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:45.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.221+0000 7ff2c4707500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.221+0000 7ff2c4707500 20 rados_obj.operate() r=0 bl.length=873 2023-10-12T18:46:45.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.221+0000 7ff2c4707500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.221+0000 7ff2c4707500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.221+0000 7ff2c4707500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.222+0000 7ff2c4707500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:45.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.222+0000 7ff2c4707500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.222+0000 7ff2c4707500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:45.228 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:46:45.228 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "ddc41caf-ed31-4401-8108-067c2eb0a43d", 2023-10-12T18:46:45.228 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default", 2023-10-12T18:46:45.228 INFO:teuthology.orchestra.run.smithi170.stdout: "domain_root": "default.rgw.meta:root", 2023-10-12T18:46:45.229 INFO:teuthology.orchestra.run.smithi170.stdout: "control_pool": "default.rgw.control", 2023-10-12T18:46:45.229 INFO:teuthology.orchestra.run.smithi170.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-12T18:46:45.229 INFO:teuthology.orchestra.run.smithi170.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-12T18:46:45.229 INFO:teuthology.orchestra.run.smithi170.stdout: "log_pool": "default.rgw.log", 2023-10-12T18:46:45.229 INFO:teuthology.orchestra.run.smithi170.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-12T18:46:45.230 INFO:teuthology.orchestra.run.smithi170.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-12T18:46:45.230 INFO:teuthology.orchestra.run.smithi170.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-12T18:46:45.230 INFO:teuthology.orchestra.run.smithi170.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-12T18:46:45.230 INFO:teuthology.orchestra.run.smithi170.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-12T18:46:45.230 INFO:teuthology.orchestra.run.smithi170.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-12T18:46:45.231 INFO:teuthology.orchestra.run.smithi170.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-12T18:46:45.231 INFO:teuthology.orchestra.run.smithi170.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-12T18:46:45.231 INFO:teuthology.orchestra.run.smithi170.stdout: "otp_pool": "default.rgw.otp", 2023-10-12T18:46:45.231 INFO:teuthology.orchestra.run.smithi170.stdout: "system_key": { 2023-10-12T18:46:45.231 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "", 2023-10-12T18:46:45.231 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "" 2023-10-12T18:46:45.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:46:45.232 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_pools": [ 2023-10-12T18:46:45.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:46:45.232 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "default-placement", 2023-10-12T18:46:45.233 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:46:45.233 INFO:teuthology.orchestra.run.smithi170.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-12T18:46:45.233 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": { 2023-10-12T18:46:45.233 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD": { 2023-10-12T18:46:45.233 INFO:teuthology.orchestra.run.smithi170.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-12T18:46:45.233 INFO:teuthology.orchestra.run.smithi170.stdout: "compression_type": "random" 2023-10-12T18:46:45.234 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:45.234 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:46:45.234 INFO:teuthology.orchestra.run.smithi170.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-12T18:46:45.234 INFO:teuthology.orchestra.run.smithi170.stdout: "index_type": 0 2023-10-12T18:46:45.235 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:45.235 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:45.235 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:46:45.235 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "", 2023-10-12T18:46:45.235 INFO:teuthology.orchestra.run.smithi170.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-12T18:46:45.236 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:46:45.237 DEBUG:tasks.util.rgw: json result: {'id': 'ddc41caf-ed31-4401-8108-067c2eb0a43d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-12T18:46:45.237 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-10-12T18:46:45.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-10-12T18:46:45.238 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-10-12T18:46:45.275 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:45.275 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:45.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.279+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.280+0000 7f36952a0500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.280+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.280+0000 7f36952a0500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.280+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.280+0000 7f36952a0500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.280+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.280+0000 7f36952a0500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:45.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.287+0000 7f36952a0500 20 RGWRados::pool_iterate: got zone_info.ddc41caf-ed31-4401-8108-067c2eb0a43d 2023-10-12T18:46:45.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.287+0000 7f36952a0500 20 RGWRados::pool_iterate: got zone_names.default 2023-10-12T18:46:45.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.287+0000 7f36952a0500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-12T18:46:45.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.287+0000 7f36952a0500 20 RGWRados::pool_iterate: got zonegroup_info.c45f860c-81fb-4480-86be-dba8a40a6a91 2023-10-12T18:46:45.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.287+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.288+0000 7f36952a0500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:45.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.288+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.288+0000 7f36952a0500 20 rados_obj.operate() r=0 bl.length=889 2023-10-12T18:46:45.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.288+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.288+0000 7f36952a0500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:45.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.288+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:45.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 10 cannot find current period zonegroup using local zonegroup 2023-10-12T18:46:45.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:45.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 zonegroup default 2023-10-12T18:46:45.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.289+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 10 Cannot find current period zone using local zone 2023-10-12T18:46:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:45.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados_obj.operate() r=0 bl.length=889 2023-10-12T18:46:45.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 zone default found 2023-10-12T18:46:45.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.290+0000 7f36952a0500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.291+0000 7f36952a0500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.291+0000 7f36952a0500 20 started sync module instance, tier type = 2023-10-12T18:46:45.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.291+0000 7f36952a0500 20 started zone id=ddc41caf-ed31-4401-8108-067c2eb0a43d (name=default) with tier type = 2023-10-12T18:46:45.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.298+0000 7f36952a0500 20 add_watcher() i=0 2023-10-12T18:46:45.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.298+0000 7f36952a0500 20 add_watcher() i=1 2023-10-12T18:46:45.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.298+0000 7f36952a0500 20 add_watcher() i=2 2023-10-12T18:46:45.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.298+0000 7f36952a0500 20 add_watcher() i=3 2023-10-12T18:46:45.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.298+0000 7f36952a0500 20 add_watcher() i=4 2023-10-12T18:46:45.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.298+0000 7f36952a0500 20 add_watcher() i=5 2023-10-12T18:46:45.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.298+0000 7f36952a0500 20 add_watcher() i=6 2023-10-12T18:46:45.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.299+0000 7f36952a0500 20 add_watcher() i=7 2023-10-12T18:46:45.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.299+0000 7f36952a0500 2 all 8 watchers are set, enabling cache 2023-10-12T18:46:45.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.300+0000 7f3651fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T18:46:45.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.300+0000 7f36952a0500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T18:46:45.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.300+0000 7f36952a0500 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T18:46:45.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.304+0000 7f36952a0500 5 note: GC not initialized 2023-10-12T18:46:45.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.305+0000 7f35c8ff9700 20 reqs_thread_entry: start 2023-10-12T18:46:45.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.305+0000 7f35be7fc700 20 reqs_thread_entry: start 2023-10-12T18:46:45.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.305+0000 7f36952a0500 20 init_complete bucket index max shards: 11 2023-10-12T18:46:45.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.305+0000 7f36952a0500 10 rgw notify: Started notification manager with: 1 workers 2023-10-12T18:46:45.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.305+0000 7f35bcff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 12 18:47:15 2023 2023-10-12T18:46:45.308 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-12T18:46:45.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.305+0000 7f35a3fff700 20 reqs_thread_entry: start 2023-10-12T18:46:45.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.313+0000 7f36952a0500 20 remove_watcher() i=0 2023-10-12T18:46:45.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.313+0000 7f36952a0500 2 removed watcher, disabling cache 2023-10-12T18:46:45.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.314+0000 7f36952a0500 20 remove_watcher() i=1 2023-10-12T18:46:45.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.315+0000 7f36952a0500 20 remove_watcher() i=2 2023-10-12T18:46:45.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.316+0000 7f36952a0500 20 remove_watcher() i=3 2023-10-12T18:46:45.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.316+0000 7f36952a0500 20 remove_watcher() i=4 2023-10-12T18:46:45.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.317+0000 7f36952a0500 20 remove_watcher() i=5 2023-10-12T18:46:45.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.318+0000 7f36952a0500 20 remove_watcher() i=6 2023-10-12T18:46:45.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.319+0000 7f36952a0500 20 remove_watcher() i=7 2023-10-12T18:46:45.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.319+0000 7f3651fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T18:46:45.324 INFO:teuthology.orchestra.run.smithi170.stdout:[] 2023-10-12T18:46:45.325 DEBUG:tasks.util.rgw: json result: [] 2023-10-12T18:46:45.326 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T18:46:45.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T18:46:45.326 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-12T18:46:45.409 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:45.409 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:45.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.413+0000 7f2a1708c500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T18:46:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.413+0000 7f2a1708c500 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T18:46:45.417 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.416+0000 7f294bfff700 20 reqs_thread_entry: start 2023-10-12T18:46:45.418 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.416+0000 7f2a1708c500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.417+0000 7f2a1708c500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:45.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.417+0000 7f2a1708c500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.417+0000 7f2a1708c500 20 rados_obj.operate() r=0 bl.length=889 2023-10-12T18:46:45.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.417+0000 7f2a1708c500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.417+0000 7f2a1708c500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:45.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.417+0000 7f2a1708c500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.417+0000 7f2a1708c500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:45.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.417+0000 7f2a1708c500 20 rados->read ofs=0 len=0 2023-10-12T18:46:45.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:45.418+0000 7f2a1708c500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:45.423 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:46:45.423 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "ddc41caf-ed31-4401-8108-067c2eb0a43d", 2023-10-12T18:46:45.423 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default", 2023-10-12T18:46:45.424 INFO:teuthology.orchestra.run.smithi170.stdout: "domain_root": "default.rgw.meta:root", 2023-10-12T18:46:45.424 INFO:teuthology.orchestra.run.smithi170.stdout: "control_pool": "default.rgw.control", 2023-10-12T18:46:45.424 INFO:teuthology.orchestra.run.smithi170.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-12T18:46:45.424 INFO:teuthology.orchestra.run.smithi170.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-12T18:46:45.424 INFO:teuthology.orchestra.run.smithi170.stdout: "log_pool": "default.rgw.log", 2023-10-12T18:46:45.425 INFO:teuthology.orchestra.run.smithi170.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-12T18:46:45.425 INFO:teuthology.orchestra.run.smithi170.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-12T18:46:45.425 INFO:teuthology.orchestra.run.smithi170.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-12T18:46:45.425 INFO:teuthology.orchestra.run.smithi170.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-12T18:46:45.425 INFO:teuthology.orchestra.run.smithi170.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-12T18:46:45.426 INFO:teuthology.orchestra.run.smithi170.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-12T18:46:45.426 INFO:teuthology.orchestra.run.smithi170.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-12T18:46:45.426 INFO:teuthology.orchestra.run.smithi170.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-12T18:46:45.426 INFO:teuthology.orchestra.run.smithi170.stdout: "otp_pool": "default.rgw.otp", 2023-10-12T18:46:45.426 INFO:teuthology.orchestra.run.smithi170.stdout: "system_key": { 2023-10-12T18:46:45.427 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "", 2023-10-12T18:46:45.427 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "" 2023-10-12T18:46:45.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:46:45.427 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_pools": [ 2023-10-12T18:46:45.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:46:45.427 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "default-placement", 2023-10-12T18:46:45.428 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:46:45.428 INFO:teuthology.orchestra.run.smithi170.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-12T18:46:45.428 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": { 2023-10-12T18:46:45.428 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD": { 2023-10-12T18:46:45.428 INFO:teuthology.orchestra.run.smithi170.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-12T18:46:45.429 INFO:teuthology.orchestra.run.smithi170.stdout: "compression_type": "random" 2023-10-12T18:46:45.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:45.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:46:45.429 INFO:teuthology.orchestra.run.smithi170.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-12T18:46:45.429 INFO:teuthology.orchestra.run.smithi170.stdout: "index_type": 0 2023-10-12T18:46:45.430 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:45.430 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:45.431 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:46:45.431 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "", 2023-10-12T18:46:45.431 INFO:teuthology.orchestra.run.smithi170.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-12T18:46:45.431 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:46:45.432 DEBUG:tasks.util.rgw: json result: {'id': 'ddc41caf-ed31-4401-8108-067c2eb0a43d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-12T18:46:45.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-10-12T18:46:45.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-10-12T18:46:45.434 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-10-12T18:46:45.536 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:45.536 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:45.543 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:45.542+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.215 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.214+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.216 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.214+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.216 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.214+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.216 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.215+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.218 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.217+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.219 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.217+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.219 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.218+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.228 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.227+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.228 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.227+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.229 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.227+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.230 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.227+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.230 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.227+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.230 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.227+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.231 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.227+0000 7fd9ee0ae500 10 cannot find current period zonegroup using local zonegroup 2023-10-12T18:46:47.231 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.227+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.231 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.228+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.231 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.228+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.231 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.228+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.232 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.228+0000 7fd9ee0ae500 10 Creating default zonegroup 2023-10-12T18:46:47.232 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.228+0000 7fd9ee0ae500 20 get_system_obj_state: rctx=0x7ffd352b4b00 obj=:.pools.avail state=0x55aa0560b810 s->prefetch_data=0 2023-10-12T18:46:47.232 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.228+0000 7fd9ee0ae500 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-12T18:46:47.237 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.235+0000 7fd9ee0ae500 10 WARNING: store->list_zones() returned r=-2 2023-10-12T18:46:47.237 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.235+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.237 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.236+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.241 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.239+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.241 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.240+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.241 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.240+0000 7fd9ee0ae500 10 could not read realm id: (2) No such file or directory 2023-10-12T18:46:47.241 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.240+0000 7fd9ee0ae500 10 WARNING: failed to set zone as default, r=-22 2023-10-12T18:46:47.242 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.240+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.242 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.240+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.243 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.242+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.245 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.243+0000 7fd9ee0ae500 20 rados_obj.operate() r=0 bl.length=873 2023-10-12T18:46:47.245 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.243+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.246 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.243+0000 7fd9ee0ae500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:47.246 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.243+0000 7fd9ee0ae500 20 zonegroup default 2023-10-12T18:46:47.246 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.243+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.246 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.243+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:47.246 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.243+0000 7fd9ee0ae500 10 Cannot find current period zone using local zone 2023-10-12T18:46:47.247 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.243+0000 7fd9ee0ae500 10 Using default name default 2023-10-12T18:46:47.247 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.243+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.247 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.244+0000 7fd9ee0ae500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:47.247 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.244+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:47.247 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.244+0000 7fd9ee0ae500 20 rados_obj.operate() r=0 bl.length=873 2023-10-12T18:46:47.248 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.244+0000 7fd9ee0ae500 20 zone default found 2023-10-12T18:46:47.248 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:47.244+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:49.222 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:49.221+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:49.223 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:49.221+0000 7fd9ee0ae500 20 rados->read ofs=0 len=0 2023-10-12T18:46:49.223 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:49.222+0000 7fd9ee0ae500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:49.223 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:49.222+0000 7fd9ee0ae500 20 started sync module instance, tier type = 2023-10-12T18:46:49.224 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:49.222+0000 7fd9ee0ae500 20 started zone id=95cc8360-69ae-4ee0-aef2-a0900cb66104 (name=default) with tier type = 2023-10-12T18:46:51.264 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.262+0000 7fd9ee0ae500 20 add_watcher() i=0 2023-10-12T18:46:51.264 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.262+0000 7fd9ee0ae500 20 add_watcher() i=1 2023-10-12T18:46:51.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.262+0000 7fd9ee0ae500 20 add_watcher() i=2 2023-10-12T18:46:51.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.262+0000 7fd9ee0ae500 20 add_watcher() i=3 2023-10-12T18:46:51.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.262+0000 7fd9ee0ae500 20 add_watcher() i=4 2023-10-12T18:46:51.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.262+0000 7fd9ee0ae500 20 add_watcher() i=5 2023-10-12T18:46:51.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.263+0000 7fd9ee0ae500 20 add_watcher() i=6 2023-10-12T18:46:51.266 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.265+0000 7fd9ee0ae500 20 add_watcher() i=7 2023-10-12T18:46:51.267 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.265+0000 7fd9ee0ae500 2 all 8 watchers are set, enabling cache 2023-10-12T18:46:51.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):409: oid=data_loggenerations_metadata not found 2023-10-12T18:46:51.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.0 2023-10-12T18:46:51.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:51.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.0 does not exist 2023-10-12T18:46:51.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.1 2023-10-12T18:46:51.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:51.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.1 does not exist 2023-10-12T18:46:51.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.2 2023-10-12T18:46:51.270 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:51.270 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.2 does not exist 2023-10-12T18:46:51.270 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.3 2023-10-12T18:46:51.270 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.266+0000 7fd9ee0ae500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:51.270 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.267+0000 7fd9ee0ae500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.3 does not exist 2023-10-12T18:46:51.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.267+0000 7fd9ee0ae500 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1206 entering 2023-10-12T18:46:51.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.268+0000 7fd9ee0ae500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-10-12T18:46:51.272 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.271+0000 7fd9aa7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T18:46:51.273 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.271+0000 7fd9ee0ae500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T18:46:51.273 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:51.271+0000 7fd9ee0ae500 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T18:46:54.224 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.221+0000 7fd9ee0ae500 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-12T18:46:54.225 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.221+0000 7fd9ee0ae500 5 note: GC not initialized 2023-10-12T18:46:54.225 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.223+0000 7fd9717fa700 20 reqs_thread_entry: start 2023-10-12T18:46:54.225 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.223+0000 7fd90affd700 20 reqs_thread_entry: start 2023-10-12T18:46:54.225 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.223+0000 7fd9ee0ae500 20 init_complete bucket index max shards: 11 2023-10-12T18:46:54.226 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.223+0000 7fd9ee0ae500 10 rgw notify: Started notification manager with: 1 workers 2023-10-12T18:46:54.226 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.223+0000 7fd9097fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 12 18:47:24 2023 2023-10-12T18:46:54.226 INFO:teuthology.orchestra.run.smithi187.stderr: 2023-10-12T18:46:54.226 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.223+0000 7fd908ff9700 20 reqs_thread_entry: start 2023-10-12T18:46:54.246 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.245+0000 7fd9ee0ae500 20 remove_watcher() i=0 2023-10-12T18:46:54.246 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.245+0000 7fd9ee0ae500 2 removed watcher, disabling cache 2023-10-12T18:46:54.249 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.248+0000 7fd9ee0ae500 20 remove_watcher() i=1 2023-10-12T18:46:54.251 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.250+0000 7fd9ee0ae500 20 remove_watcher() i=2 2023-10-12T18:46:54.254 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.253+0000 7fd9ee0ae500 20 remove_watcher() i=3 2023-10-12T18:46:54.256 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.255+0000 7fd9ee0ae500 20 remove_watcher() i=4 2023-10-12T18:46:54.259 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.258+0000 7fd9ee0ae500 20 remove_watcher() i=5 2023-10-12T18:46:54.260 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.259+0000 7fd9ee0ae500 20 remove_watcher() i=6 2023-10-12T18:46:54.261 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.260+0000 7fd9ee0ae500 20 remove_watcher() i=7 2023-10-12T18:46:54.262 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.261+0000 7fd9aa7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T18:46:54.265 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2023-10-12T18:46:54.267 DEBUG:tasks.util.rgw: json result: [] 2023-10-12T18:46:54.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T18:46:54.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T18:46:54.268 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-12T18:46:54.315 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:54.315 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:54.321 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.320+0000 7f0462379500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T18:46:54.321 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.320+0000 7f0462379500 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T18:46:54.324 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.323+0000 7f03977fe700 20 reqs_thread_entry: start 2023-10-12T18:46:54.325 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.323+0000 7f0462379500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.327 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.324+0000 7f0462379500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:54.327 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.324+0000 7f0462379500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.327 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.324+0000 7f0462379500 20 rados_obj.operate() r=0 bl.length=873 2023-10-12T18:46:54.327 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.324+0000 7f0462379500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.328 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.325+0000 7f0462379500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.328 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.325+0000 7f0462379500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.328 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.325+0000 7f0462379500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:54.328 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.325+0000 7f0462379500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.329 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.326+0000 7f0462379500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:54.331 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:46:54.331 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "95cc8360-69ae-4ee0-aef2-a0900cb66104", 2023-10-12T18:46:54.331 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "default", 2023-10-12T18:46:54.331 INFO:teuthology.orchestra.run.smithi187.stdout: "domain_root": "default.rgw.meta:root", 2023-10-12T18:46:54.332 INFO:teuthology.orchestra.run.smithi187.stdout: "control_pool": "default.rgw.control", 2023-10-12T18:46:54.332 INFO:teuthology.orchestra.run.smithi187.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-12T18:46:54.332 INFO:teuthology.orchestra.run.smithi187.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-12T18:46:54.332 INFO:teuthology.orchestra.run.smithi187.stdout: "log_pool": "default.rgw.log", 2023-10-12T18:46:54.332 INFO:teuthology.orchestra.run.smithi187.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-12T18:46:54.333 INFO:teuthology.orchestra.run.smithi187.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-12T18:46:54.333 INFO:teuthology.orchestra.run.smithi187.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-12T18:46:54.333 INFO:teuthology.orchestra.run.smithi187.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-12T18:46:54.333 INFO:teuthology.orchestra.run.smithi187.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-12T18:46:54.333 INFO:teuthology.orchestra.run.smithi187.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-12T18:46:54.333 INFO:teuthology.orchestra.run.smithi187.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-12T18:46:54.334 INFO:teuthology.orchestra.run.smithi187.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-12T18:46:54.334 INFO:teuthology.orchestra.run.smithi187.stdout: "otp_pool": "default.rgw.otp", 2023-10-12T18:46:54.334 INFO:teuthology.orchestra.run.smithi187.stdout: "system_key": { 2023-10-12T18:46:54.334 INFO:teuthology.orchestra.run.smithi187.stdout: "access_key": "", 2023-10-12T18:46:54.334 INFO:teuthology.orchestra.run.smithi187.stdout: "secret_key": "" 2023-10-12T18:46:54.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:46:54.335 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_pools": [ 2023-10-12T18:46:54.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:46:54.335 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "default-placement", 2023-10-12T18:46:54.335 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:46:54.336 INFO:teuthology.orchestra.run.smithi187.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-12T18:46:54.336 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": { 2023-10-12T18:46:54.336 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD": { 2023-10-12T18:46:54.336 INFO:teuthology.orchestra.run.smithi187.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-12T18:46:54.336 INFO:teuthology.orchestra.run.smithi187.stdout: "compression_type": "random" 2023-10-12T18:46:54.337 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:46:54.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:46:54.337 INFO:teuthology.orchestra.run.smithi187.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-12T18:46:54.337 INFO:teuthology.orchestra.run.smithi187.stdout: "index_type": 0 2023-10-12T18:46:54.337 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:46:54.337 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:46:54.338 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:46:54.338 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "", 2023-10-12T18:46:54.338 INFO:teuthology.orchestra.run.smithi187.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-12T18:46:54.338 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:46:54.339 DEBUG:tasks.util.rgw: json result: {'id': '95cc8360-69ae-4ee0-aef2-a0900cb66104', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-12T18:46:54.340 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-10-12T18:46:54.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-10-12T18:46:54.340 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-10-12T18:46:54.378 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:54.378 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:54.383 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.382+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.384 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.383+0000 7f3d93116500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.384 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.383+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.385 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.383+0000 7f3d93116500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.385 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.383+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.385 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.383+0000 7f3d93116500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.385 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.383+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.386 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.383+0000 7f3d93116500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:54.392 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.390+0000 7f3d93116500 20 RGWRados::pool_iterate: got zone_info.95cc8360-69ae-4ee0-aef2-a0900cb66104 2023-10-12T18:46:54.392 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.390+0000 7f3d93116500 20 RGWRados::pool_iterate: got zonegroup_info.32405982-e162-4b66-a6ce-abae8cf009fd 2023-10-12T18:46:54.392 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.390+0000 7f3d93116500 20 RGWRados::pool_iterate: got zone_names.default 2023-10-12T18:46:54.393 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.390+0000 7f3d93116500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-12T18:46:54.393 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.390+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.393 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.391+0000 7f3d93116500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:54.393 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.391+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.396 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.391+0000 7f3d93116500 20 rados_obj.operate() r=0 bl.length=889 2023-10-12T18:46:54.396 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.391+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.396 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.391+0000 7f3d93116500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:54.396 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.391+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.397 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.392+0000 7f3d93116500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:54.397 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.392+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.397 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.392+0000 7f3d93116500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.397 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.392+0000 7f3d93116500 10 cannot find current period zonegroup using local zonegroup 2023-10-12T18:46:54.397 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.392+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.398 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.392+0000 7f3d93116500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.398 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.392+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.398 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.392+0000 7f3d93116500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:54.398 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.392+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.399 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:54.399 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 zonegroup default 2023-10-12T18:46:54.399 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.399 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.399 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 10 Cannot find current period zone using local zone 2023-10-12T18:46:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 rados_obj.operate() r=0 bl.length=889 2023-10-12T18:46:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.393+0000 7f3d93116500 20 zone default found 2023-10-12T18:46:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.394+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.394+0000 7f3d93116500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.394+0000 7f3d93116500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.394+0000 7f3d93116500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.402 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.394+0000 7f3d93116500 20 started sync module instance, tier type = 2023-10-12T18:46:54.402 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.394+0000 7f3d93116500 20 started zone id=95cc8360-69ae-4ee0-aef2-a0900cb66104 (name=default) with tier type = 2023-10-12T18:46:54.403 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.402+0000 7f3d93116500 20 add_watcher() i=0 2023-10-12T18:46:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.402+0000 7f3d93116500 20 add_watcher() i=1 2023-10-12T18:46:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.402+0000 7f3d93116500 20 add_watcher() i=2 2023-10-12T18:46:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.402+0000 7f3d93116500 20 add_watcher() i=3 2023-10-12T18:46:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.402+0000 7f3d93116500 20 add_watcher() i=4 2023-10-12T18:46:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.402+0000 7f3d93116500 20 add_watcher() i=5 2023-10-12T18:46:54.405 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.402+0000 7f3d93116500 20 add_watcher() i=6 2023-10-12T18:46:54.405 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.403+0000 7f3d93116500 20 add_watcher() i=7 2023-10-12T18:46:54.405 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.403+0000 7f3d93116500 2 all 8 watchers are set, enabling cache 2023-10-12T18:46:54.406 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.404+0000 7f3d4ffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T18:46:54.406 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.404+0000 7f3d93116500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T18:46:54.406 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.404+0000 7f3d93116500 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T18:46:54.409 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.408+0000 7f3d93116500 5 note: GC not initialized 2023-10-12T18:46:54.410 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.409+0000 7f3cbeffd700 20 reqs_thread_entry: start 2023-10-12T18:46:54.411 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.409+0000 7f3cbcff9700 20 reqs_thread_entry: start 2023-10-12T18:46:54.411 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.409+0000 7f3d93116500 20 init_complete bucket index max shards: 11 2023-10-12T18:46:54.411 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.409+0000 7f3d93116500 10 rgw notify: Started notification manager with: 1 workers 2023-10-12T18:46:54.411 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.409+0000 7f3cb2ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 12 18:47:24 2023 2023-10-12T18:46:54.412 INFO:teuthology.orchestra.run.smithi187.stderr: 2023-10-12T18:46:54.412 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.409+0000 7f3cb27fc700 20 reqs_thread_entry: start 2023-10-12T18:46:54.418 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.417+0000 7f3d93116500 20 remove_watcher() i=0 2023-10-12T18:46:54.418 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.417+0000 7f3d93116500 2 removed watcher, disabling cache 2023-10-12T18:46:54.419 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.418+0000 7f3d93116500 20 remove_watcher() i=1 2023-10-12T18:46:54.419 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.418+0000 7f3d93116500 20 remove_watcher() i=2 2023-10-12T18:46:54.421 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.420+0000 7f3d93116500 20 remove_watcher() i=3 2023-10-12T18:46:54.422 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.421+0000 7f3d93116500 20 remove_watcher() i=4 2023-10-12T18:46:54.423 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.422+0000 7f3d93116500 20 remove_watcher() i=5 2023-10-12T18:46:54.423 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.422+0000 7f3d93116500 20 remove_watcher() i=6 2023-10-12T18:46:54.424 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.423+0000 7f3d93116500 20 remove_watcher() i=7 2023-10-12T18:46:54.425 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.424+0000 7f3d4ffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T18:46:54.428 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2023-10-12T18:46:54.429 DEBUG:tasks.util.rgw: json result: [] 2023-10-12T18:46:54.430 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T18:46:54.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T18:46:54.431 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-12T18:46:54.511 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:54.512 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:54.517 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.515+0000 7fae3aa10500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T18:46:54.517 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.515+0000 7fae3aa10500 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T18:46:54.520 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.519+0000 7fae3aa10500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.520 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.519+0000 7fad6effd700 20 reqs_thread_entry: start 2023-10-12T18:46:54.521 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.519+0000 7fae3aa10500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:54.521 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.520+0000 7fae3aa10500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.522 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.520+0000 7fae3aa10500 20 rados_obj.operate() r=0 bl.length=889 2023-10-12T18:46:54.522 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.520+0000 7fae3aa10500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.522 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.520+0000 7fae3aa10500 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T18:46:54.522 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.520+0000 7fae3aa10500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.522 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.520+0000 7fae3aa10500 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T18:46:54.523 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.520+0000 7fae3aa10500 20 rados->read ofs=0 len=0 2023-10-12T18:46:54.523 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:46:54.521+0000 7fae3aa10500 20 rados_obj.operate() r=0 bl.length=358 2023-10-12T18:46:54.525 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:46:54.526 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "95cc8360-69ae-4ee0-aef2-a0900cb66104", 2023-10-12T18:46:54.526 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "default", 2023-10-12T18:46:54.526 INFO:teuthology.orchestra.run.smithi187.stdout: "domain_root": "default.rgw.meta:root", 2023-10-12T18:46:54.526 INFO:teuthology.orchestra.run.smithi187.stdout: "control_pool": "default.rgw.control", 2023-10-12T18:46:54.526 INFO:teuthology.orchestra.run.smithi187.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-12T18:46:54.527 INFO:teuthology.orchestra.run.smithi187.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-12T18:46:54.527 INFO:teuthology.orchestra.run.smithi187.stdout: "log_pool": "default.rgw.log", 2023-10-12T18:46:54.527 INFO:teuthology.orchestra.run.smithi187.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-12T18:46:54.527 INFO:teuthology.orchestra.run.smithi187.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-12T18:46:54.527 INFO:teuthology.orchestra.run.smithi187.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-12T18:46:54.528 INFO:teuthology.orchestra.run.smithi187.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-12T18:46:54.528 INFO:teuthology.orchestra.run.smithi187.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-12T18:46:54.528 INFO:teuthology.orchestra.run.smithi187.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-12T18:46:54.528 INFO:teuthology.orchestra.run.smithi187.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-12T18:46:54.528 INFO:teuthology.orchestra.run.smithi187.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-12T18:46:54.529 INFO:teuthology.orchestra.run.smithi187.stdout: "otp_pool": "default.rgw.otp", 2023-10-12T18:46:54.529 INFO:teuthology.orchestra.run.smithi187.stdout: "system_key": { 2023-10-12T18:46:54.529 INFO:teuthology.orchestra.run.smithi187.stdout: "access_key": "", 2023-10-12T18:46:54.529 INFO:teuthology.orchestra.run.smithi187.stdout: "secret_key": "" 2023-10-12T18:46:54.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:46:54.529 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_pools": [ 2023-10-12T18:46:54.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:46:54.530 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "default-placement", 2023-10-12T18:46:54.530 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:46:54.530 INFO:teuthology.orchestra.run.smithi187.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-12T18:46:54.530 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": { 2023-10-12T18:46:54.531 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD": { 2023-10-12T18:46:54.531 INFO:teuthology.orchestra.run.smithi187.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-12T18:46:54.531 INFO:teuthology.orchestra.run.smithi187.stdout: "compression_type": "random" 2023-10-12T18:46:54.531 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:46:54.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:46:54.532 INFO:teuthology.orchestra.run.smithi187.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-12T18:46:54.532 INFO:teuthology.orchestra.run.smithi187.stdout: "index_type": 0 2023-10-12T18:46:54.532 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:46:54.532 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:46:54.532 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:46:54.532 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "", 2023-10-12T18:46:54.533 INFO:teuthology.orchestra.run.smithi187.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-12T18:46:54.533 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:46:54.534 DEBUG:tasks.util.rgw: json result: {'id': '95cc8360-69ae-4ee0-aef2-a0900cb66104', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-12T18:46:54.534 INFO:tasks.rgw:Starting rgw... 2023-10-12T18:46:54.535 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-10-12T18:46:54.535 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-12T18:46:54.535 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-12T18:46:54.536 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-12T18:46:54.539 INFO:tasks.rgw.c1.client.0:Started 2023-10-12T18:46:54.539 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-10-12T18:46:54.539 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-12T18:46:54.539 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T18:46:54.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T18:46:54.542 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T18:46:54.542 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-10-12T18:46:54.542 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-12T18:46:54.543 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-12T18:46:54.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-12T18:46:54.576 INFO:tasks.rgw.c2.client.0:Started 2023-10-12T18:46:54.577 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-10-12T18:46:54.577 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-12T18:46:54.577 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-12T18:46:54.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-12T18:46:54.580 INFO:tasks.rgw.c2.client.1:Started 2023-10-12T18:46:54.580 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi170.front.sepia.ceph.com:8000/ 2023-10-12T18:46:54.580 DEBUG:teuthology.orchestra.run.smithi170:> curl http://smithi170.front.sepia.ceph.com:8000/ 2023-10-12T18:46:54.616 INFO:teuthology.orchestra.run.smithi170.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:46:54.616 INFO:teuthology.orchestra.run.smithi170.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:46:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T18:46:54.619 INFO:teuthology.orchestra.run.smithi170.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi170.front.sepia.ceph.com port 8000: Connection refused 2023-10-12T18:46:55.619 DEBUG:teuthology.orchestra.run.smithi170:> curl http://smithi170.front.sepia.ceph.com:8000/ 2023-10-12T18:46:55.639 INFO:teuthology.orchestra.run.smithi170.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:46:55.639 INFO:teuthology.orchestra.run.smithi170.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:46:55.640 INFO:teuthology.orchestra.run.smithi170.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T18:46:55.641 INFO:teuthology.orchestra.run.smithi170.stdout:anonymous 2023-10-12T18:46:55.641 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi170.front.sepia.ceph.com:8001/ 2023-10-12T18:46:55.642 DEBUG:teuthology.orchestra.run.smithi170:> curl http://smithi170.front.sepia.ceph.com:8001/ 2023-10-12T18:46:55.700 INFO:teuthology.orchestra.run.smithi170.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:46:55.701 INFO:teuthology.orchestra.run.smithi170.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:46:55.701 INFO:teuthology.orchestra.run.smithi170.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T18:46:55.702 INFO:teuthology.orchestra.run.smithi170.stdout:anonymous 2023-10-12T18:46:55.702 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi187.front.sepia.ceph.com:8000/ 2023-10-12T18:46:55.703 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8000/ 2023-10-12T18:46:55.737 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:46:55.737 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:46:55.737 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T18:46:55.738 INFO:teuthology.orchestra.run.smithi187.stdout:anonymous 2023-10-12T18:46:55.739 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi187.front.sepia.ceph.com:8001/ 2023-10-12T18:46:55.739 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8001/ 2023-10-12T18:46:55.798 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:46:55.798 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:46:55.798 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T18:46:55.799 INFO:teuthology.orchestra.run.smithi187.stdout:anonymous 2023-10-12T18:46:55.800 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-10-12T18:46:55.859 INFO:tasks.rgw_multisite:creating realm.. 2023-10-12T18:46:55.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:55.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:55.861 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:46:55.899 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:55.899 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:55.918 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:46:55.919 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:46:55.919 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "test-realm", 2023-10-12T18:46:55.919 INFO:teuthology.orchestra.run.smithi170.stdout: "current_period": "e1332b00-f761-4c62-a5ed-b8b60a8676fd", 2023-10-12T18:46:55.919 INFO:teuthology.orchestra.run.smithi170.stdout: "epoch": 1 2023-10-12T18:46:55.919 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:46:55.920 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-10-12T18:46:55.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:55.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:55.922 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:46:56.000 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:56.000 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:56.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:46:56.009+0000 7fa05ba0b500 -1 old format 2023-10-12T18:46:56.016 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:46:56.016 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "a", 2023-10-12T18:46:56.016 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a", 2023-10-12T18:46:56.016 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "", 2023-10-12T18:46:56.016 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "true", 2023-10-12T18:46:56.017 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T18:46:56.017 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T18:46:56.017 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:46:56.017 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T18:46:56.017 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T18:46:56.018 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "", 2023-10-12T18:46:56.018 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [], 2023-10-12T18:46:56.018 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [], 2023-10-12T18:46:56.018 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "", 2023-10-12T18:46:56.018 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:46:56.019 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T18:46:56.019 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T18:46:56.019 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:56.019 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:46:56.020 INFO:tasks.rgw_multisite:creating master zone.. 2023-10-12T18:46:56.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--endpoints', 'http://smithi170.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:56.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--endpoints', 'http://smithi170.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:56.021 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key 29UBGG6NS65BV4RP --secret JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5 --endpoints http://smithi170.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:46:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:56.108 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:56.135 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:46:56.136 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:46:56.136 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a1", 2023-10-12T18:46:56.136 INFO:teuthology.orchestra.run.smithi170.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-12T18:46:56.136 INFO:teuthology.orchestra.run.smithi170.stdout: "control_pool": "a1.rgw.control", 2023-10-12T18:46:56.136 INFO:teuthology.orchestra.run.smithi170.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-12T18:46:56.137 INFO:teuthology.orchestra.run.smithi170.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-12T18:46:56.137 INFO:teuthology.orchestra.run.smithi170.stdout: "log_pool": "a1.rgw.log", 2023-10-12T18:46:56.137 INFO:teuthology.orchestra.run.smithi170.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-12T18:46:56.137 INFO:teuthology.orchestra.run.smithi170.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-12T18:46:56.137 INFO:teuthology.orchestra.run.smithi170.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-12T18:46:56.138 INFO:teuthology.orchestra.run.smithi170.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-12T18:46:56.138 INFO:teuthology.orchestra.run.smithi170.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-12T18:46:56.138 INFO:teuthology.orchestra.run.smithi170.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-12T18:46:56.138 INFO:teuthology.orchestra.run.smithi170.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-12T18:46:56.138 INFO:teuthology.orchestra.run.smithi170.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-12T18:46:56.139 INFO:teuthology.orchestra.run.smithi170.stdout: "otp_pool": "a1.rgw.otp", 2023-10-12T18:46:56.139 INFO:teuthology.orchestra.run.smithi170.stdout: "system_key": { 2023-10-12T18:46:56.139 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T18:46:56.139 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T18:46:56.139 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:46:56.139 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_pools": [ 2023-10-12T18:46:56.140 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:46:56.140 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "default-placement", 2023-10-12T18:46:56.140 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:46:56.140 INFO:teuthology.orchestra.run.smithi170.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-12T18:46:56.140 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": { 2023-10-12T18:46:56.141 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD": { 2023-10-12T18:46:56.141 INFO:teuthology.orchestra.run.smithi170.stdout: "data_pool": "a1.rgw.buckets.data" 2023-10-12T18:46:56.141 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:56.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:46:56.141 INFO:teuthology.orchestra.run.smithi170.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-12T18:46:56.141 INFO:teuthology.orchestra.run.smithi170.stdout: "index_type": 0 2023-10-12T18:46:56.142 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:56.142 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:56.142 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:46:56.142 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:46:56.142 INFO:teuthology.orchestra.run.smithi170.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-12T18:46:56.143 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:46:56.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-10-12T18:46:57.146 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'a1.rgw.buckets.data' created 2023-10-12T18:46:57.161 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-10-12T18:46:58.171 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-10-12T18:46:58.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:58.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:58.184 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:46:58.228 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:58.228 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:46:58.242 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:46:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:46:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a1", 2023-10-12T18:46:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-12T18:46:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "control_pool": "a1.rgw.control", 2023-10-12T18:46:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-12T18:46:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-12T18:46:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "log_pool": "a1.rgw.log", 2023-10-12T18:46:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-12T18:46:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-12T18:46:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-12T18:46:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-12T18:46:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-12T18:46:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-12T18:46:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-12T18:46:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-12T18:46:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "otp_pool": "a1.rgw.otp", 2023-10-12T18:46:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "system_key": { 2023-10-12T18:46:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T18:46:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T18:46:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:46:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_pools": [ 2023-10-12T18:46:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:46:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "default-placement", 2023-10-12T18:46:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:46:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-12T18:46:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": { 2023-10-12T18:46:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD": { 2023-10-12T18:46:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "data_pool": "a1.rgw.buckets.data", 2023-10-12T18:46:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "compression_type": "random" 2023-10-12T18:46:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:46:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-12T18:46:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "index_type": 0 2023-10-12T18:46:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:46:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:46:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "", 2023-10-12T18:46:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-12T18:46:58.250 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:46:58.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:58.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:46:58.252 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:46:58.292 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:46:58.292 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:05.223 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:47:05.224 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:05.224 INFO:teuthology.orchestra.run.smithi170.stdout: "epoch": 1, 2023-10-12T18:47:05.224 INFO:teuthology.orchestra.run.smithi170.stdout: "predecessor_uuid": "e1332b00-f761-4c62-a5ed-b8b60a8676fd", 2023-10-12T18:47:05.224 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": [], 2023-10-12T18:47:05.224 INFO:teuthology.orchestra.run.smithi170.stdout: "period_map": { 2023-10-12T18:47:05.225 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:05.225 INFO:teuthology.orchestra.run.smithi170.stdout: "zonegroups": [ 2023-10-12T18:47:05.225 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:05.225 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "a", 2023-10-12T18:47:05.225 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a", 2023-10-12T18:47:05.226 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "", 2023-10-12T18:47:05.226 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "true", 2023-10-12T18:47:05.226 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T18:47:05.226 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T18:47:05.226 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:05.227 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T18:47:05.227 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T18:47:05.227 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:05.227 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [ 2023-10-12T18:47:05.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:05.228 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:05.228 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a1", 2023-10-12T18:47:05.228 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T18:47:05.228 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T18:47:05.228 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:05.229 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T18:47:05.229 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "false", 2023-10-12T18:47:05.229 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:05.229 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T18:47:05.229 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T18:47:05.230 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T18:47:05.230 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T18:47:05.230 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T18:47:05.230 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:05.231 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:05.231 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [ 2023-10-12T18:47:05.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:05.231 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default-placement", 2023-10-12T18:47:05.231 INFO:teuthology.orchestra.run.smithi170.stdout: "tags": [], 2023-10-12T18:47:05.232 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": [ 2023-10-12T18:47:05.232 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD" 2023-10-12T18:47:05.232 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:47:05.232 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:05.232 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:05.233 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "default-placement", 2023-10-12T18:47:05.233 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:05.233 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T18:47:05.233 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T18:47:05.233 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:05.234 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:05.234 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:05.234 INFO:teuthology.orchestra.run.smithi170.stdout: "short_zone_ids": [ 2023-10-12T18:47:05.234 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:05.234 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:05.235 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 2601847591 2023-10-12T18:47:05.235 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:05.235 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:47:05.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:05.235 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zonegroup": "a", 2023-10-12T18:47:05.236 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:05.236 INFO:teuthology.orchestra.run.smithi170.stdout: "period_config": { 2023-10-12T18:47:05.236 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_quota": { 2023-10-12T18:47:05.236 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T18:47:05.236 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T18:47:05.237 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T18:47:05.237 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T18:47:05.237 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T18:47:05.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:05.237 INFO:teuthology.orchestra.run.smithi170.stdout: "user_quota": { 2023-10-12T18:47:05.237 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T18:47:05.238 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T18:47:05.238 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T18:47:05.238 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T18:47:05.239 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T18:47:05.239 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:05.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:05.239 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:05.239 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_name": "test-realm", 2023-10-12T18:47:05.239 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:47:05.241 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:47:05.242 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-12T18:47:05.243 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T18:47:05.284 INFO:tasks.rgw.c1.client.0.smithi170.stdout:2023-10-12T18:47:05.283+0000 7fa8deefc700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 89244) UID: 0 2023-10-12T18:47:05.284 INFO:tasks.rgw.c1.client.0.smithi170.stdout:2023-10-12T18:47:05.283+0000 7fa8f77be5c0 -1 shutting down 2023-10-12T18:47:08.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-12T18:47:11.344 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-12T18:47:11.345 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-12T18:47:11.346 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-12T18:47:11.350 INFO:tasks.rgw.c1.client.0:Started 2023-10-12T18:47:11.351 DEBUG:teuthology.orchestra.run.smithi170:> curl http://smithi170.front.sepia.ceph.com:8000 2023-10-12T18:47:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T18:47:11.373 INFO:teuthology.orchestra.run.smithi170.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:47:11.373 INFO:teuthology.orchestra.run.smithi170.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:47:11.374 INFO:teuthology.orchestra.run.smithi170.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi170.front.sepia.ceph.com port 8000: Connection refused 2023-10-12T18:47:12.375 DEBUG:teuthology.orchestra.run.smithi170:> curl http://smithi170.front.sepia.ceph.com:8000 2023-10-12T18:47:12.395 INFO:teuthology.orchestra.run.smithi170.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:47:12.395 INFO:teuthology.orchestra.run.smithi170.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:47:12.395 INFO:teuthology.orchestra.run.smithi170.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T18:47:12.396 INFO:teuthology.orchestra.run.smithi170.stdout:anonymous 2023-10-12T18:47:12.396 INFO:tasks.rgw_multisite:creating admin user.. 2023-10-12T18:47:12.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:12.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:12.398 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 29UBGG6NS65BV4RP --secret JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:12.485 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:12.485 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:12.522 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:47:12.522 INFO:teuthology.orchestra.run.smithi170.stdout: "user_id": "realm-admin", 2023-10-12T18:47:12.522 INFO:teuthology.orchestra.run.smithi170.stdout: "display_name": "Realm Admin", 2023-10-12T18:47:12.522 INFO:teuthology.orchestra.run.smithi170.stdout: "email": "", 2023-10-12T18:47:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: "suspended": 0, 2023-10-12T18:47:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: "max_buckets": 1000, 2023-10-12T18:47:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: "subusers": [], 2023-10-12T18:47:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: "keys": [ 2023-10-12T18:47:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: "user": "realm-admin", 2023-10-12T18:47:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T18:47:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T18:47:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: "swift_keys": [], 2023-10-12T18:47:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": [], 2023-10-12T18:47:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: "op_mask": "read, write, delete", 2023-10-12T18:47:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: "system": "true", 2023-10-12T18:47:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "", 2023-10-12T18:47:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: "default_storage_class": "", 2023-10-12T18:47:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_tags": [], 2023-10-12T18:47:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_quota": { 2023-10-12T18:47:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T18:47:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T18:47:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T18:47:12.527 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T18:47:12.527 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T18:47:12.527 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: "user_quota": { 2023-10-12T18:47:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T18:47:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T18:47:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T18:47:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T18:47:12.529 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T18:47:12.529 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:12.529 INFO:teuthology.orchestra.run.smithi170.stdout: "temp_url_keys": [], 2023-10-12T18:47:12.529 INFO:teuthology.orchestra.run.smithi170.stdout: "type": "rgw", 2023-10-12T18:47:12.529 INFO:teuthology.orchestra.run.smithi170.stdout: "mfa_ids": [] 2023-10-12T18:47:12.529 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:47:12.530 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:47:12.533 INFO:tasks.rgw_multisite:creating zone.. 2023-10-12T18:47:12.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--endpoints', 'http://smithi170.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:12.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--endpoints', 'http://smithi170.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:12.535 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key 29UBGG6NS65BV4RP --secret JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5 --endpoints http://smithi170.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:12.580 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:12.580 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:12.606 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:47:12.606 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T18:47:12.606 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a2", 2023-10-12T18:47:12.606 INFO:teuthology.orchestra.run.smithi170.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-12T18:47:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: "control_pool": "a2.rgw.control", 2023-10-12T18:47:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-12T18:47:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-12T18:47:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: "log_pool": "a2.rgw.log", 2023-10-12T18:47:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-12T18:47:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-12T18:47:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-12T18:47:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-12T18:47:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-12T18:47:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-12T18:47:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-12T18:47:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-12T18:47:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: "otp_pool": "a2.rgw.otp", 2023-10-12T18:47:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: "system_key": { 2023-10-12T18:47:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T18:47:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T18:47:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_pools": [ 2023-10-12T18:47:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "default-placement", 2023-10-12T18:47:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:47:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-12T18:47:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": { 2023-10-12T18:47:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD": { 2023-10-12T18:47:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: "data_pool": "a2.rgw.buckets.data" 2023-10-12T18:47:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-12T18:47:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: "index_type": 0 2023-10-12T18:47:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:12.613 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:12.613 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:12.613 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:12.613 INFO:teuthology.orchestra.run.smithi170.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-12T18:47:12.613 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:47:12.615 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-10-12T18:47:13.177 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'a2.rgw.buckets.data' created 2023-10-12T18:47:13.195 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-10-12T18:47:14.206 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-10-12T18:47:14.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:14.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:14.222 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:14.268 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:14.268 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:14.282 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:47:14.283 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T18:47:14.283 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a2", 2023-10-12T18:47:14.283 INFO:teuthology.orchestra.run.smithi170.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-12T18:47:14.283 INFO:teuthology.orchestra.run.smithi170.stdout: "control_pool": "a2.rgw.control", 2023-10-12T18:47:14.283 INFO:teuthology.orchestra.run.smithi170.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-12T18:47:14.284 INFO:teuthology.orchestra.run.smithi170.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-12T18:47:14.284 INFO:teuthology.orchestra.run.smithi170.stdout: "log_pool": "a2.rgw.log", 2023-10-12T18:47:14.284 INFO:teuthology.orchestra.run.smithi170.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-12T18:47:14.284 INFO:teuthology.orchestra.run.smithi170.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-12T18:47:14.285 INFO:teuthology.orchestra.run.smithi170.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-12T18:47:14.285 INFO:teuthology.orchestra.run.smithi170.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-12T18:47:14.285 INFO:teuthology.orchestra.run.smithi170.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-12T18:47:14.286 INFO:teuthology.orchestra.run.smithi170.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-12T18:47:14.286 INFO:teuthology.orchestra.run.smithi170.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-12T18:47:14.286 INFO:teuthology.orchestra.run.smithi170.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-12T18:47:14.287 INFO:teuthology.orchestra.run.smithi170.stdout: "otp_pool": "a2.rgw.otp", 2023-10-12T18:47:14.287 INFO:teuthology.orchestra.run.smithi170.stdout: "system_key": { 2023-10-12T18:47:14.287 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T18:47:14.287 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T18:47:14.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:14.288 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_pools": [ 2023-10-12T18:47:14.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:14.288 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "default-placement", 2023-10-12T18:47:14.288 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:47:14.288 INFO:teuthology.orchestra.run.smithi170.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-12T18:47:14.289 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": { 2023-10-12T18:47:14.289 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD": { 2023-10-12T18:47:14.289 INFO:teuthology.orchestra.run.smithi170.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-12T18:47:14.289 INFO:teuthology.orchestra.run.smithi170.stdout: "compression_type": "random" 2023-10-12T18:47:14.289 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:14.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:14.290 INFO:teuthology.orchestra.run.smithi170.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-12T18:47:14.290 INFO:teuthology.orchestra.run.smithi170.stdout: "index_type": 0 2023-10-12T18:47:14.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:14.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:14.291 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:14.291 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "", 2023-10-12T18:47:14.291 INFO:teuthology.orchestra.run.smithi170.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-12T18:47:14.291 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:47:14.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:14.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:14.293 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:14.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-12T18:47:14.374 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:14.374 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:21.246 INFO:teuthology.orchestra.run.smithi170.stderr:Sending period to new master zone cc69b071-ffc5-4662-8e68-2301691e9014 2023-10-12T18:47:21.282 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:47:21.282 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:21.282 INFO:teuthology.orchestra.run.smithi170.stdout: "epoch": 2, 2023-10-12T18:47:21.283 INFO:teuthology.orchestra.run.smithi170.stdout: "predecessor_uuid": "e1332b00-f761-4c62-a5ed-b8b60a8676fd", 2023-10-12T18:47:21.283 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": [], 2023-10-12T18:47:21.283 INFO:teuthology.orchestra.run.smithi170.stdout: "period_map": { 2023-10-12T18:47:21.283 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:21.283 INFO:teuthology.orchestra.run.smithi170.stdout: "zonegroups": [ 2023-10-12T18:47:21.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:21.284 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "a", 2023-10-12T18:47:21.284 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a", 2023-10-12T18:47:21.284 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "", 2023-10-12T18:47:21.284 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "true", 2023-10-12T18:47:21.285 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T18:47:21.285 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T18:47:21.285 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:21.285 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T18:47:21.285 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T18:47:21.286 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:21.286 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [ 2023-10-12T18:47:21.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:21.286 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T18:47:21.286 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a2", 2023-10-12T18:47:21.286 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T18:47:21.287 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8001" 2023-10-12T18:47:21.287 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:21.287 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T18:47:21.287 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T18:47:21.287 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:21.288 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T18:47:21.288 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T18:47:21.288 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T18:47:21.288 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T18:47:21.288 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T18:47:21.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:21.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:21.289 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:21.289 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a1", 2023-10-12T18:47:21.289 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T18:47:21.290 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T18:47:21.290 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:21.290 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T18:47:21.291 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T18:47:21.291 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:21.291 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T18:47:21.291 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T18:47:21.291 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T18:47:21.292 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T18:47:21.292 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T18:47:21.292 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:21.292 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:21.292 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [ 2023-10-12T18:47:21.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:21.293 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default-placement", 2023-10-12T18:47:21.293 INFO:teuthology.orchestra.run.smithi170.stdout: "tags": [], 2023-10-12T18:47:21.293 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": [ 2023-10-12T18:47:21.293 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD" 2023-10-12T18:47:21.294 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:47:21.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:21.294 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:21.294 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "default-placement", 2023-10-12T18:47:21.294 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:21.295 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T18:47:21.295 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T18:47:21.295 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:21.295 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:21.295 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:47:21.296 INFO:teuthology.orchestra.run.smithi170.stdout: "short_zone_ids": [ 2023-10-12T18:47:21.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:21.296 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T18:47:21.296 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 3365504037 2023-10-12T18:47:21.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:21.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:47:21.297 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:21.297 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 2601847591 2023-10-12T18:47:21.297 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:21.297 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:47:21.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:21.298 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zonegroup": "a", 2023-10-12T18:47:21.298 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:21.298 INFO:teuthology.orchestra.run.smithi170.stdout: "period_config": { 2023-10-12T18:47:21.298 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_quota": { 2023-10-12T18:47:21.298 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T18:47:21.299 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T18:47:21.299 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T18:47:21.299 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T18:47:21.299 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T18:47:21.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:21.300 INFO:teuthology.orchestra.run.smithi170.stdout: "user_quota": { 2023-10-12T18:47:21.300 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T18:47:21.300 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T18:47:21.300 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T18:47:21.300 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T18:47:21.301 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T18:47:21.301 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:47:21.301 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:47:21.301 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:21.301 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_name": "test-realm", 2023-10-12T18:47:21.301 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:47:21.302 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:47:21.303 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T18:47:21.303 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T18:47:21.345 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T18:47:21.344+0000 7fc583cd7700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 89266) UID: 0 2023-10-12T18:47:21.345 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T18:47:21.344+0000 7fc59c5995c0 -1 shutting down 2023-10-12T18:47:25.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T18:47:27.406 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T18:47:27.406 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T18:47:27.407 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T18:47:27.410 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T18:47:27.410 DEBUG:teuthology.orchestra.run.smithi170:> curl http://smithi170.front.sepia.ceph.com:8001 2023-10-12T18:47:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T18:47:27.433 INFO:teuthology.orchestra.run.smithi170.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:47:27.433 INFO:teuthology.orchestra.run.smithi170.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:47:27.433 INFO:teuthology.orchestra.run.smithi170.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi170.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T18:47:28.434 DEBUG:teuthology.orchestra.run.smithi170:> curl http://smithi170.front.sepia.ceph.com:8001 2023-10-12T18:47:28.455 INFO:teuthology.orchestra.run.smithi170.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:47:28.456 INFO:teuthology.orchestra.run.smithi170.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:47:28.456 INFO:teuthology.orchestra.run.smithi170.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T18:47:28.457 INFO:teuthology.orchestra.run.smithi170.stdout:anonymous 2023-10-12T18:47:28.457 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-12T18:47:28.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi170.front.sepia.ceph.com:8000', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:28.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi170.front.sepia.ceph.com:8000', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:28.458 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi170.front.sepia.ceph.com:8000 --access-key 29UBGG6NS65BV4RP --secret JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:28.510 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:28.510 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:28.543 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:47:28.543 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:28.543 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "test-realm", 2023-10-12T18:47:28.545 INFO:teuthology.orchestra.run.smithi187.stdout: "current_period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:28.545 INFO:teuthology.orchestra.run.smithi187.stdout: "epoch": 2 2023-10-12T18:47:28.545 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:47:28.547 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-10-12T18:47:28.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:28.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:28.548 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:28.589 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:28.589 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:28.599 INFO:teuthology.orchestra.run.smithi187.stderr:2023-10-12T18:47:28.598+0000 7fae047e5500 -1 old format 2023-10-12T18:47:28.605 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:47:28.606 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "b", 2023-10-12T18:47:28.606 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b", 2023-10-12T18:47:28.606 INFO:teuthology.orchestra.run.smithi187.stdout: "api_name": "", 2023-10-12T18:47:28.606 INFO:teuthology.orchestra.run.smithi187.stdout: "is_master": "false", 2023-10-12T18:47:28.606 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:28.607 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T18:47:28.607 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:28.607 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames": [], 2023-10-12T18:47:28.607 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames_s3website": [], 2023-10-12T18:47:28.607 INFO:teuthology.orchestra.run.smithi187.stdout: "master_zone": "", 2023-10-12T18:47:28.608 INFO:teuthology.orchestra.run.smithi187.stdout: "zones": [], 2023-10-12T18:47:28.608 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_targets": [], 2023-10-12T18:47:28.608 INFO:teuthology.orchestra.run.smithi187.stdout: "default_placement": "", 2023-10-12T18:47:28.608 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:28.608 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_policy": { 2023-10-12T18:47:28.609 INFO:teuthology.orchestra.run.smithi187.stdout: "groups": [] 2023-10-12T18:47:28.609 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:28.609 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:47:28.610 INFO:tasks.rgw_multisite:creating zone.. 2023-10-12T18:47:28.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--endpoints', 'http://smithi187.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:28.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--endpoints', 'http://smithi187.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:28.611 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key 29UBGG6NS65BV4RP --secret JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5 --endpoints http://smithi187.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:28.697 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:28.698 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:28.726 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:47:28.726 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T18:47:28.726 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b1", 2023-10-12T18:47:28.726 INFO:teuthology.orchestra.run.smithi187.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-12T18:47:28.727 INFO:teuthology.orchestra.run.smithi187.stdout: "control_pool": "b1.rgw.control", 2023-10-12T18:47:28.727 INFO:teuthology.orchestra.run.smithi187.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-12T18:47:28.727 INFO:teuthology.orchestra.run.smithi187.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-12T18:47:28.727 INFO:teuthology.orchestra.run.smithi187.stdout: "log_pool": "b1.rgw.log", 2023-10-12T18:47:28.727 INFO:teuthology.orchestra.run.smithi187.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-12T18:47:28.728 INFO:teuthology.orchestra.run.smithi187.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-12T18:47:28.728 INFO:teuthology.orchestra.run.smithi187.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-12T18:47:28.728 INFO:teuthology.orchestra.run.smithi187.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-12T18:47:28.728 INFO:teuthology.orchestra.run.smithi187.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-12T18:47:28.728 INFO:teuthology.orchestra.run.smithi187.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-12T18:47:28.729 INFO:teuthology.orchestra.run.smithi187.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-12T18:47:28.729 INFO:teuthology.orchestra.run.smithi187.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-12T18:47:28.729 INFO:teuthology.orchestra.run.smithi187.stdout: "otp_pool": "b1.rgw.otp", 2023-10-12T18:47:28.729 INFO:teuthology.orchestra.run.smithi187.stdout: "system_key": { 2023-10-12T18:47:28.729 INFO:teuthology.orchestra.run.smithi187.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T18:47:28.730 INFO:teuthology.orchestra.run.smithi187.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T18:47:28.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:28.730 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_pools": [ 2023-10-12T18:47:28.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:28.730 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "default-placement", 2023-10-12T18:47:28.731 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:47:28.731 INFO:teuthology.orchestra.run.smithi187.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-12T18:47:28.731 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": { 2023-10-12T18:47:28.731 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD": { 2023-10-12T18:47:28.731 INFO:teuthology.orchestra.run.smithi187.stdout: "data_pool": "b1.rgw.buckets.data" 2023-10-12T18:47:28.732 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:28.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:28.732 INFO:teuthology.orchestra.run.smithi187.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-12T18:47:28.732 INFO:teuthology.orchestra.run.smithi187.stdout: "index_type": 0 2023-10-12T18:47:28.733 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:28.733 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:28.733 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:28.733 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:28.734 INFO:teuthology.orchestra.run.smithi187.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-12T18:47:28.734 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:47:28.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-10-12T18:47:29.280 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'b1.rgw.buckets.data' created 2023-10-12T18:47:29.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-10-12T18:47:30.302 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-10-12T18:47:30.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:30.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:30.321 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:30.366 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:30.366 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:30.380 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:47:30.381 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T18:47:30.381 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b1", 2023-10-12T18:47:30.381 INFO:teuthology.orchestra.run.smithi187.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-12T18:47:30.381 INFO:teuthology.orchestra.run.smithi187.stdout: "control_pool": "b1.rgw.control", 2023-10-12T18:47:30.382 INFO:teuthology.orchestra.run.smithi187.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-12T18:47:30.382 INFO:teuthology.orchestra.run.smithi187.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-12T18:47:30.382 INFO:teuthology.orchestra.run.smithi187.stdout: "log_pool": "b1.rgw.log", 2023-10-12T18:47:30.382 INFO:teuthology.orchestra.run.smithi187.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-12T18:47:30.383 INFO:teuthology.orchestra.run.smithi187.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-12T18:47:30.383 INFO:teuthology.orchestra.run.smithi187.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-12T18:47:30.383 INFO:teuthology.orchestra.run.smithi187.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-12T18:47:30.383 INFO:teuthology.orchestra.run.smithi187.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-12T18:47:30.383 INFO:teuthology.orchestra.run.smithi187.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-12T18:47:30.383 INFO:teuthology.orchestra.run.smithi187.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-12T18:47:30.384 INFO:teuthology.orchestra.run.smithi187.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-12T18:47:30.384 INFO:teuthology.orchestra.run.smithi187.stdout: "otp_pool": "b1.rgw.otp", 2023-10-12T18:47:30.384 INFO:teuthology.orchestra.run.smithi187.stdout: "system_key": { 2023-10-12T18:47:30.384 INFO:teuthology.orchestra.run.smithi187.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T18:47:30.384 INFO:teuthology.orchestra.run.smithi187.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T18:47:30.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:30.385 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_pools": [ 2023-10-12T18:47:30.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:30.385 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "default-placement", 2023-10-12T18:47:30.385 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:47:30.386 INFO:teuthology.orchestra.run.smithi187.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-12T18:47:30.386 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": { 2023-10-12T18:47:30.386 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD": { 2023-10-12T18:47:30.386 INFO:teuthology.orchestra.run.smithi187.stdout: "data_pool": "b1.rgw.buckets.data", 2023-10-12T18:47:30.387 INFO:teuthology.orchestra.run.smithi187.stdout: "compression_type": "random" 2023-10-12T18:47:30.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:30.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:30.387 INFO:teuthology.orchestra.run.smithi187.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-12T18:47:30.387 INFO:teuthology.orchestra.run.smithi187.stdout: "index_type": 0 2023-10-12T18:47:30.388 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:30.388 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:30.388 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:30.388 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "", 2023-10-12T18:47:30.388 INFO:teuthology.orchestra.run.smithi187.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-12T18:47:30.389 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:47:30.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:30.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:30.391 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:30.431 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:30.431 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:31.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T18:47:37.349 INFO:teuthology.orchestra.run.smithi187.stderr:Sending period to new master zone cc69b071-ffc5-4662-8e68-2301691e9014 2023-10-12T18:47:37.382 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:47:37.382 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:37.383 INFO:teuthology.orchestra.run.smithi187.stdout: "epoch": 3, 2023-10-12T18:47:37.383 INFO:teuthology.orchestra.run.smithi187.stdout: "predecessor_uuid": "e1332b00-f761-4c62-a5ed-b8b60a8676fd", 2023-10-12T18:47:37.383 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_status": [], 2023-10-12T18:47:37.383 INFO:teuthology.orchestra.run.smithi187.stdout: "period_map": { 2023-10-12T18:47:37.383 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:37.384 INFO:teuthology.orchestra.run.smithi187.stdout: "zonegroups": [ 2023-10-12T18:47:37.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.384 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "a", 2023-10-12T18:47:37.384 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "a", 2023-10-12T18:47:37.384 INFO:teuthology.orchestra.run.smithi187.stdout: "api_name": "", 2023-10-12T18:47:37.385 INFO:teuthology.orchestra.run.smithi187.stdout: "is_master": "true", 2023-10-12T18:47:37.385 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:37.385 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T18:47:37.385 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.385 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames": [], 2023-10-12T18:47:37.386 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames_s3website": [], 2023-10-12T18:47:37.386 INFO:teuthology.orchestra.run.smithi187.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:37.386 INFO:teuthology.orchestra.run.smithi187.stdout: "zones": [ 2023-10-12T18:47:37.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.387 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T18:47:37.387 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "a2", 2023-10-12T18:47:37.387 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:37.387 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi170.front.sepia.ceph.com:8001" 2023-10-12T18:47:37.388 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.388 INFO:teuthology.orchestra.run.smithi187.stdout: "log_meta": "false", 2023-10-12T18:47:37.388 INFO:teuthology.orchestra.run.smithi187.stdout: "log_data": "true", 2023-10-12T18:47:37.388 INFO:teuthology.orchestra.run.smithi187.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:37.388 INFO:teuthology.orchestra.run.smithi187.stdout: "read_only": "false", 2023-10-12T18:47:37.389 INFO:teuthology.orchestra.run.smithi187.stdout: "tier_type": "", 2023-10-12T18:47:37.389 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from_all": "true", 2023-10-12T18:47:37.389 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from": [], 2023-10-12T18:47:37.389 INFO:teuthology.orchestra.run.smithi187.stdout: "redirect_zone": "" 2023-10-12T18:47:37.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:37.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.390 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:37.390 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "a1", 2023-10-12T18:47:37.391 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:37.391 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T18:47:37.391 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.391 INFO:teuthology.orchestra.run.smithi187.stdout: "log_meta": "false", 2023-10-12T18:47:37.392 INFO:teuthology.orchestra.run.smithi187.stdout: "log_data": "true", 2023-10-12T18:47:37.392 INFO:teuthology.orchestra.run.smithi187.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:37.392 INFO:teuthology.orchestra.run.smithi187.stdout: "read_only": "false", 2023-10-12T18:47:37.393 INFO:teuthology.orchestra.run.smithi187.stdout: "tier_type": "", 2023-10-12T18:47:37.393 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from_all": "true", 2023-10-12T18:47:37.393 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from": [], 2023-10-12T18:47:37.393 INFO:teuthology.orchestra.run.smithi187.stdout: "redirect_zone": "" 2023-10-12T18:47:37.394 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:37.394 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.394 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_targets": [ 2023-10-12T18:47:37.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.395 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "default-placement", 2023-10-12T18:47:37.395 INFO:teuthology.orchestra.run.smithi187.stdout: "tags": [], 2023-10-12T18:47:37.395 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": [ 2023-10-12T18:47:37.395 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD" 2023-10-12T18:47:37.395 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:47:37.397 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:37.397 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.397 INFO:teuthology.orchestra.run.smithi187.stdout: "default_placement": "default-placement", 2023-10-12T18:47:37.397 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:37.397 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_policy": { 2023-10-12T18:47:37.398 INFO:teuthology.orchestra.run.smithi187.stdout: "groups": [] 2023-10-12T18:47:37.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:37.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:37.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.399 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "b", 2023-10-12T18:47:37.399 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b", 2023-10-12T18:47:37.399 INFO:teuthology.orchestra.run.smithi187.stdout: "api_name": "", 2023-10-12T18:47:37.399 INFO:teuthology.orchestra.run.smithi187.stdout: "is_master": "false", 2023-10-12T18:47:37.399 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:37.400 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T18:47:37.400 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.400 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames": [], 2023-10-12T18:47:37.400 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames_s3website": [], 2023-10-12T18:47:37.400 INFO:teuthology.orchestra.run.smithi187.stdout: "master_zone": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T18:47:37.401 INFO:teuthology.orchestra.run.smithi187.stdout: "zones": [ 2023-10-12T18:47:37.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.401 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T18:47:37.402 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b1", 2023-10-12T18:47:37.402 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:37.402 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T18:47:37.402 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.402 INFO:teuthology.orchestra.run.smithi187.stdout: "log_meta": "false", 2023-10-12T18:47:37.403 INFO:teuthology.orchestra.run.smithi187.stdout: "log_data": "false", 2023-10-12T18:47:37.403 INFO:teuthology.orchestra.run.smithi187.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:37.403 INFO:teuthology.orchestra.run.smithi187.stdout: "read_only": "false", 2023-10-12T18:47:37.403 INFO:teuthology.orchestra.run.smithi187.stdout: "tier_type": "", 2023-10-12T18:47:37.404 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from_all": "true", 2023-10-12T18:47:37.404 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from": [], 2023-10-12T18:47:37.404 INFO:teuthology.orchestra.run.smithi187.stdout: "redirect_zone": "" 2023-10-12T18:47:37.404 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:37.404 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.405 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_targets": [ 2023-10-12T18:47:37.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.405 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "default-placement", 2023-10-12T18:47:37.405 INFO:teuthology.orchestra.run.smithi187.stdout: "tags": [], 2023-10-12T18:47:37.405 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": [ 2023-10-12T18:47:37.406 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD" 2023-10-12T18:47:37.406 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:47:37.406 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:37.406 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.406 INFO:teuthology.orchestra.run.smithi187.stdout: "default_placement": "default-placement", 2023-10-12T18:47:37.407 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:37.408 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_policy": { 2023-10-12T18:47:37.408 INFO:teuthology.orchestra.run.smithi187.stdout: "groups": [] 2023-10-12T18:47:37.409 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:37.409 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:37.410 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:37.410 INFO:teuthology.orchestra.run.smithi187.stdout: "short_zone_ids": [ 2023-10-12T18:47:37.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.410 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T18:47:37.410 INFO:teuthology.orchestra.run.smithi187.stdout: "val": 3365504037 2023-10-12T18:47:37.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:37.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.411 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T18:47:37.411 INFO:teuthology.orchestra.run.smithi187.stdout: "val": 20676319 2023-10-12T18:47:37.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:37.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:37.412 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:37.412 INFO:teuthology.orchestra.run.smithi187.stdout: "val": 2601847591 2023-10-12T18:47:37.413 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:37.413 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:47:37.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:37.414 INFO:teuthology.orchestra.run.smithi187.stdout: "master_zonegroup": "a", 2023-10-12T18:47:37.414 INFO:teuthology.orchestra.run.smithi187.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:37.414 INFO:teuthology.orchestra.run.smithi187.stdout: "period_config": { 2023-10-12T18:47:37.414 INFO:teuthology.orchestra.run.smithi187.stdout: "bucket_quota": { 2023-10-12T18:47:37.414 INFO:teuthology.orchestra.run.smithi187.stdout: "enabled": false, 2023-10-12T18:47:37.415 INFO:teuthology.orchestra.run.smithi187.stdout: "check_on_raw": false, 2023-10-12T18:47:37.415 INFO:teuthology.orchestra.run.smithi187.stdout: "max_size": -1, 2023-10-12T18:47:37.415 INFO:teuthology.orchestra.run.smithi187.stdout: "max_size_kb": 0, 2023-10-12T18:47:37.415 INFO:teuthology.orchestra.run.smithi187.stdout: "max_objects": -1 2023-10-12T18:47:37.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:37.416 INFO:teuthology.orchestra.run.smithi187.stdout: "user_quota": { 2023-10-12T18:47:37.416 INFO:teuthology.orchestra.run.smithi187.stdout: "enabled": false, 2023-10-12T18:47:37.416 INFO:teuthology.orchestra.run.smithi187.stdout: "check_on_raw": false, 2023-10-12T18:47:37.416 INFO:teuthology.orchestra.run.smithi187.stdout: "max_size": -1, 2023-10-12T18:47:37.416 INFO:teuthology.orchestra.run.smithi187.stdout: "max_size_kb": 0, 2023-10-12T18:47:37.417 INFO:teuthology.orchestra.run.smithi187.stdout: "max_objects": -1 2023-10-12T18:47:37.417 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:37.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:37.418 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:37.418 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_name": "test-realm", 2023-10-12T18:47:37.418 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:47:37.419 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:47:37.419 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-12T18:47:37.420 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T18:47:37.449 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2023-10-12T18:47:37.448+0000 7f8d157de700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 89248) UID: 0 2023-10-12T18:47:37.450 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2023-10-12T18:47:37.448+0000 7f8d2e0a05c0 -1 shutting down 2023-10-12T18:47:43.315 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-12T18:47:43.521 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-12T18:47:43.522 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-12T18:47:43.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-12T18:47:43.526 INFO:tasks.rgw.c2.client.0:Started 2023-10-12T18:47:43.526 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8000 2023-10-12T18:47:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T18:47:43.548 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:47:43.548 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:47:43.549 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi187.front.sepia.ceph.com port 8000: Connection refused 2023-10-12T18:47:44.550 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8000 2023-10-12T18:47:44.569 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:47:44.570 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:47:44.571 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T18:47:44.573 INFO:teuthology.orchestra.run.smithi187.stdout:anonymous 2023-10-12T18:47:44.573 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-12T18:47:44.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi170.front.sepia.ceph.com:8000', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--url', 'http://smithi170.front.sepia.ceph.com:8000', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:44.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi170.front.sepia.ceph.com:8000', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--url', 'http://smithi170.front.sepia.ceph.com:8000', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:44.575 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi170.front.sepia.ceph.com:8000 --access-key 29UBGG6NS65BV4RP --secret JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5 --url http://smithi170.front.sepia.ceph.com:8000 --access-key 29UBGG6NS65BV4RP --secret JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:44.663 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:44.663 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:44.685 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2023-10-12T18:47:44.683+0000 7f74eb548700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.833c22c1-dbac-4225-ba1b-f4ce971ab379.control err=-107 2023-10-12T18:47:44.697 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:47:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "test-realm", 2023-10-12T18:47:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "current_period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "epoch": 2 2023-10-12T18:47:44.699 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:47:44.701 INFO:tasks.rgw_multisite:creating zone.. 2023-10-12T18:47:44.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--endpoints', 'http://smithi187.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:44.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', '29UBGG6NS65BV4RP', '--secret', 'JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5', '--endpoints', 'http://smithi187.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:44.702 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key 29UBGG6NS65BV4RP --secret JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5 --endpoints http://smithi187.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:44.748 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:44.748 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:44.774 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:47:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T18:47:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b2", 2023-10-12T18:47:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-12T18:47:44.776 INFO:teuthology.orchestra.run.smithi187.stdout: "control_pool": "b2.rgw.control", 2023-10-12T18:47:44.776 INFO:teuthology.orchestra.run.smithi187.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-12T18:47:44.776 INFO:teuthology.orchestra.run.smithi187.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-12T18:47:44.777 INFO:teuthology.orchestra.run.smithi187.stdout: "log_pool": "b2.rgw.log", 2023-10-12T18:47:44.777 INFO:teuthology.orchestra.run.smithi187.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-12T18:47:44.778 INFO:teuthology.orchestra.run.smithi187.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-12T18:47:44.778 INFO:teuthology.orchestra.run.smithi187.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-12T18:47:44.778 INFO:teuthology.orchestra.run.smithi187.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-12T18:47:44.779 INFO:teuthology.orchestra.run.smithi187.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-12T18:47:44.779 INFO:teuthology.orchestra.run.smithi187.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-12T18:47:44.779 INFO:teuthology.orchestra.run.smithi187.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-12T18:47:44.780 INFO:teuthology.orchestra.run.smithi187.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-12T18:47:44.780 INFO:teuthology.orchestra.run.smithi187.stdout: "otp_pool": "b2.rgw.otp", 2023-10-12T18:47:44.781 INFO:teuthology.orchestra.run.smithi187.stdout: "system_key": { 2023-10-12T18:47:44.781 INFO:teuthology.orchestra.run.smithi187.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T18:47:44.781 INFO:teuthology.orchestra.run.smithi187.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T18:47:44.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:44.782 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_pools": [ 2023-10-12T18:47:44.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:44.783 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "default-placement", 2023-10-12T18:47:44.783 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:47:44.783 INFO:teuthology.orchestra.run.smithi187.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-12T18:47:44.784 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": { 2023-10-12T18:47:44.785 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD": { 2023-10-12T18:47:44.785 INFO:teuthology.orchestra.run.smithi187.stdout: "data_pool": "b2.rgw.buckets.data" 2023-10-12T18:47:44.785 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:44.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:44.786 INFO:teuthology.orchestra.run.smithi187.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-12T18:47:44.787 INFO:teuthology.orchestra.run.smithi187.stdout: "index_type": 0 2023-10-12T18:47:44.787 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:44.787 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:44.788 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:44.788 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:44.788 INFO:teuthology.orchestra.run.smithi187.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-12T18:47:44.789 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:47:44.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-10-12T18:47:45.328 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'b2.rgw.buckets.data' created 2023-10-12T18:47:45.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-10-12T18:47:46.351 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-10-12T18:47:46.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:46.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:46.372 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:46.433 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:47:46.434 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T18:47:46.434 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b2", 2023-10-12T18:47:46.434 INFO:teuthology.orchestra.run.smithi187.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-12T18:47:46.435 INFO:teuthology.orchestra.run.smithi187.stdout: "control_pool": "b2.rgw.control", 2023-10-12T18:47:46.435 INFO:teuthology.orchestra.run.smithi187.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-12T18:47:46.435 INFO:teuthology.orchestra.run.smithi187.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-12T18:47:46.435 INFO:teuthology.orchestra.run.smithi187.stdout: "log_pool": "b2.rgw.log", 2023-10-12T18:47:46.436 INFO:teuthology.orchestra.run.smithi187.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-12T18:47:46.436 INFO:teuthology.orchestra.run.smithi187.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-12T18:47:46.436 INFO:teuthology.orchestra.run.smithi187.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-12T18:47:46.436 INFO:teuthology.orchestra.run.smithi187.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-12T18:47:46.436 INFO:teuthology.orchestra.run.smithi187.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-12T18:47:46.437 INFO:teuthology.orchestra.run.smithi187.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-12T18:47:46.437 INFO:teuthology.orchestra.run.smithi187.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-12T18:47:46.437 INFO:teuthology.orchestra.run.smithi187.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-12T18:47:46.437 INFO:teuthology.orchestra.run.smithi187.stdout: "otp_pool": "b2.rgw.otp", 2023-10-12T18:47:46.437 INFO:teuthology.orchestra.run.smithi187.stdout: "system_key": { 2023-10-12T18:47:46.438 INFO:teuthology.orchestra.run.smithi187.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T18:47:46.438 INFO:teuthology.orchestra.run.smithi187.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T18:47:46.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:46.438 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_pools": [ 2023-10-12T18:47:46.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:46.439 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "default-placement", 2023-10-12T18:47:46.439 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:47:46.439 INFO:teuthology.orchestra.run.smithi187.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-12T18:47:46.439 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": { 2023-10-12T18:47:46.439 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD": { 2023-10-12T18:47:46.440 INFO:teuthology.orchestra.run.smithi187.stdout: "data_pool": "b2.rgw.buckets.data", 2023-10-12T18:47:46.440 INFO:teuthology.orchestra.run.smithi187.stdout: "compression_type": "random" 2023-10-12T18:47:46.440 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:46.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:46.441 INFO:teuthology.orchestra.run.smithi187.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-12T18:47:46.441 INFO:teuthology.orchestra.run.smithi187.stdout: "index_type": 0 2023-10-12T18:47:46.441 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:46.441 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:46.442 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:46.442 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "", 2023-10-12T18:47:46.442 INFO:teuthology.orchestra.run.smithi187.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-12T18:47:46.442 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:47:46.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:46.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:47:46.444 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T18:47:46.484 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:47:46.485 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:47:49.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-12T18:47:53.399 INFO:teuthology.orchestra.run.smithi187.stderr:Sending period to new master zone cc69b071-ffc5-4662-8e68-2301691e9014 2023-10-12T18:47:53.442 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:47:53.443 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:53.443 INFO:teuthology.orchestra.run.smithi187.stdout: "epoch": 4, 2023-10-12T18:47:53.443 INFO:teuthology.orchestra.run.smithi187.stdout: "predecessor_uuid": "e1332b00-f761-4c62-a5ed-b8b60a8676fd", 2023-10-12T18:47:53.443 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_status": [], 2023-10-12T18:47:53.444 INFO:teuthology.orchestra.run.smithi187.stdout: "period_map": { 2023-10-12T18:47:53.444 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:47:53.444 INFO:teuthology.orchestra.run.smithi187.stdout: "zonegroups": [ 2023-10-12T18:47:53.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.444 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "a", 2023-10-12T18:47:53.444 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "a", 2023-10-12T18:47:53.445 INFO:teuthology.orchestra.run.smithi187.stdout: "api_name": "", 2023-10-12T18:47:53.445 INFO:teuthology.orchestra.run.smithi187.stdout: "is_master": "true", 2023-10-12T18:47:53.445 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:53.445 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T18:47:53.445 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.446 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames": [], 2023-10-12T18:47:53.446 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames_s3website": [], 2023-10-12T18:47:53.446 INFO:teuthology.orchestra.run.smithi187.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:53.446 INFO:teuthology.orchestra.run.smithi187.stdout: "zones": [ 2023-10-12T18:47:53.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.447 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T18:47:53.447 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "a2", 2023-10-12T18:47:53.447 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:53.447 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi170.front.sepia.ceph.com:8001" 2023-10-12T18:47:53.447 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.448 INFO:teuthology.orchestra.run.smithi187.stdout: "log_meta": "false", 2023-10-12T18:47:53.448 INFO:teuthology.orchestra.run.smithi187.stdout: "log_data": "true", 2023-10-12T18:47:53.448 INFO:teuthology.orchestra.run.smithi187.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:53.448 INFO:teuthology.orchestra.run.smithi187.stdout: "read_only": "false", 2023-10-12T18:47:53.448 INFO:teuthology.orchestra.run.smithi187.stdout: "tier_type": "", 2023-10-12T18:47:53.448 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from_all": "true", 2023-10-12T18:47:53.449 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from": [], 2023-10-12T18:47:53.449 INFO:teuthology.orchestra.run.smithi187.stdout: "redirect_zone": "" 2023-10-12T18:47:53.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:53.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.449 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:53.450 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "a1", 2023-10-12T18:47:53.450 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:53.450 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T18:47:53.450 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.450 INFO:teuthology.orchestra.run.smithi187.stdout: "log_meta": "false", 2023-10-12T18:47:53.451 INFO:teuthology.orchestra.run.smithi187.stdout: "log_data": "true", 2023-10-12T18:47:53.451 INFO:teuthology.orchestra.run.smithi187.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:53.451 INFO:teuthology.orchestra.run.smithi187.stdout: "read_only": "false", 2023-10-12T18:47:53.451 INFO:teuthology.orchestra.run.smithi187.stdout: "tier_type": "", 2023-10-12T18:47:53.451 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from_all": "true", 2023-10-12T18:47:53.452 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from": [], 2023-10-12T18:47:53.452 INFO:teuthology.orchestra.run.smithi187.stdout: "redirect_zone": "" 2023-10-12T18:47:53.452 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:53.453 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.453 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_targets": [ 2023-10-12T18:47:53.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.453 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "default-placement", 2023-10-12T18:47:53.453 INFO:teuthology.orchestra.run.smithi187.stdout: "tags": [], 2023-10-12T18:47:53.454 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": [ 2023-10-12T18:47:53.454 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD" 2023-10-12T18:47:53.454 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:47:53.454 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:53.454 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.455 INFO:teuthology.orchestra.run.smithi187.stdout: "default_placement": "default-placement", 2023-10-12T18:47:53.455 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:53.455 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_policy": { 2023-10-12T18:47:53.455 INFO:teuthology.orchestra.run.smithi187.stdout: "groups": [] 2023-10-12T18:47:53.455 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:53.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:53.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.456 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "b", 2023-10-12T18:47:53.456 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b", 2023-10-12T18:47:53.456 INFO:teuthology.orchestra.run.smithi187.stdout: "api_name": "", 2023-10-12T18:47:53.456 INFO:teuthology.orchestra.run.smithi187.stdout: "is_master": "false", 2023-10-12T18:47:53.457 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:53.457 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T18:47:53.457 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.457 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames": [], 2023-10-12T18:47:53.457 INFO:teuthology.orchestra.run.smithi187.stdout: "hostnames_s3website": [], 2023-10-12T18:47:53.458 INFO:teuthology.orchestra.run.smithi187.stdout: "master_zone": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T18:47:53.458 INFO:teuthology.orchestra.run.smithi187.stdout: "zones": [ 2023-10-12T18:47:53.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.458 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T18:47:53.458 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b2", 2023-10-12T18:47:53.458 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:53.459 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi187.front.sepia.ceph.com:8001" 2023-10-12T18:47:53.459 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.459 INFO:teuthology.orchestra.run.smithi187.stdout: "log_meta": "false", 2023-10-12T18:47:53.459 INFO:teuthology.orchestra.run.smithi187.stdout: "log_data": "true", 2023-10-12T18:47:53.459 INFO:teuthology.orchestra.run.smithi187.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:53.460 INFO:teuthology.orchestra.run.smithi187.stdout: "read_only": "false", 2023-10-12T18:47:53.460 INFO:teuthology.orchestra.run.smithi187.stdout: "tier_type": "", 2023-10-12T18:47:53.460 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from_all": "true", 2023-10-12T18:47:53.460 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from": [], 2023-10-12T18:47:53.460 INFO:teuthology.orchestra.run.smithi187.stdout: "redirect_zone": "" 2023-10-12T18:47:53.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:53.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.461 INFO:teuthology.orchestra.run.smithi187.stdout: "id": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T18:47:53.461 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "b1", 2023-10-12T18:47:53.461 INFO:teuthology.orchestra.run.smithi187.stdout: "endpoints": [ 2023-10-12T18:47:53.461 INFO:teuthology.orchestra.run.smithi187.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T18:47:53.462 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.462 INFO:teuthology.orchestra.run.smithi187.stdout: "log_meta": "false", 2023-10-12T18:47:53.462 INFO:teuthology.orchestra.run.smithi187.stdout: "log_data": "true", 2023-10-12T18:47:53.462 INFO:teuthology.orchestra.run.smithi187.stdout: "bucket_index_max_shards": 11, 2023-10-12T18:47:53.462 INFO:teuthology.orchestra.run.smithi187.stdout: "read_only": "false", 2023-10-12T18:47:53.464 INFO:teuthology.orchestra.run.smithi187.stdout: "tier_type": "", 2023-10-12T18:47:53.464 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from_all": "true", 2023-10-12T18:47:53.464 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_from": [], 2023-10-12T18:47:53.464 INFO:teuthology.orchestra.run.smithi187.stdout: "redirect_zone": "" 2023-10-12T18:47:53.464 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:53.465 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.465 INFO:teuthology.orchestra.run.smithi187.stdout: "placement_targets": [ 2023-10-12T18:47:53.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.465 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "default-placement", 2023-10-12T18:47:53.465 INFO:teuthology.orchestra.run.smithi187.stdout: "tags": [], 2023-10-12T18:47:53.466 INFO:teuthology.orchestra.run.smithi187.stdout: "storage_classes": [ 2023-10-12T18:47:53.466 INFO:teuthology.orchestra.run.smithi187.stdout: "STANDARD" 2023-10-12T18:47:53.466 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:47:53.466 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:53.466 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.467 INFO:teuthology.orchestra.run.smithi187.stdout: "default_placement": "default-placement", 2023-10-12T18:47:53.467 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:53.467 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_policy": { 2023-10-12T18:47:53.467 INFO:teuthology.orchestra.run.smithi187.stdout: "groups": [] 2023-10-12T18:47:53.467 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:53.468 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:53.468 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2023-10-12T18:47:53.468 INFO:teuthology.orchestra.run.smithi187.stdout: "short_zone_ids": [ 2023-10-12T18:47:53.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.468 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T18:47:53.468 INFO:teuthology.orchestra.run.smithi187.stdout: "val": 3365504037 2023-10-12T18:47:53.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:53.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.469 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T18:47:53.469 INFO:teuthology.orchestra.run.smithi187.stdout: "val": 259669302 2023-10-12T18:47:53.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:53.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.470 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T18:47:53.470 INFO:teuthology.orchestra.run.smithi187.stdout: "val": 20676319 2023-10-12T18:47:53.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:53.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:47:53.471 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:53.471 INFO:teuthology.orchestra.run.smithi187.stdout: "val": 2601847591 2023-10-12T18:47:53.471 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:53.471 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:47:53.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:53.472 INFO:teuthology.orchestra.run.smithi187.stdout: "master_zonegroup": "a", 2023-10-12T18:47:53.472 INFO:teuthology.orchestra.run.smithi187.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T18:47:53.472 INFO:teuthology.orchestra.run.smithi187.stdout: "period_config": { 2023-10-12T18:47:53.472 INFO:teuthology.orchestra.run.smithi187.stdout: "bucket_quota": { 2023-10-12T18:47:53.472 INFO:teuthology.orchestra.run.smithi187.stdout: "enabled": false, 2023-10-12T18:47:53.473 INFO:teuthology.orchestra.run.smithi187.stdout: "check_on_raw": false, 2023-10-12T18:47:53.473 INFO:teuthology.orchestra.run.smithi187.stdout: "max_size": -1, 2023-10-12T18:47:53.473 INFO:teuthology.orchestra.run.smithi187.stdout: "max_size_kb": 0, 2023-10-12T18:47:53.473 INFO:teuthology.orchestra.run.smithi187.stdout: "max_objects": -1 2023-10-12T18:47:53.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:53.473 INFO:teuthology.orchestra.run.smithi187.stdout: "user_quota": { 2023-10-12T18:47:53.474 INFO:teuthology.orchestra.run.smithi187.stdout: "enabled": false, 2023-10-12T18:47:53.474 INFO:teuthology.orchestra.run.smithi187.stdout: "check_on_raw": false, 2023-10-12T18:47:53.474 INFO:teuthology.orchestra.run.smithi187.stdout: "max_size": -1, 2023-10-12T18:47:53.474 INFO:teuthology.orchestra.run.smithi187.stdout: "max_size_kb": 0, 2023-10-12T18:47:53.474 INFO:teuthology.orchestra.run.smithi187.stdout: "max_objects": -1 2023-10-12T18:47:53.475 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:47:53.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:47:53.475 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T18:47:53.475 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_name": "test-realm", 2023-10-12T18:47:53.475 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:47:53.476 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:47:53.476 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-12T18:47:53.477 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T18:47:53.505 INFO:tasks.rgw.c2.client.1.smithi187.stdout:2023-10-12T18:47:53.504+0000 7ff4c80bf700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 89252) UID: 0 2023-10-12T18:47:53.506 INFO:tasks.rgw.c2.client.1.smithi187.stdout:2023-10-12T18:47:53.504+0000 7ff4e09815c0 -1 shutting down 2023-10-12T18:47:54.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-12T18:47:59.579 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-12T18:47:59.579 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-12T18:47:59.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-12T18:47:59.583 INFO:tasks.rgw.c2.client.1:Started 2023-10-12T18:47:59.583 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8001 2023-10-12T18:47:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T18:47:59.606 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:47:59.606 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:47:59.606 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi187.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T18:48:00.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-12T18:48:00.607 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8001 2023-10-12T18:48:00.627 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T18:48:00.627 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2023-10-12T18:48:00.628 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T18:48:00.629 INFO:teuthology.orchestra.run.smithi187.stdout:anonymous 2023-10-12T18:48:00.629 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-10-12T18:48:00.629 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-10-12T18:48:00.651 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-10-12T18:48:00.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:48:00.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T18:48:00.653 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T18:48:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:00.750 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:00.750 INFO:teuthology.orchestra.run.smithi170.stdout: "user_id": "rgw-multisite-test-user", 2023-10-12T18:48:00.750 INFO:teuthology.orchestra.run.smithi170.stdout: "display_name": "Multisite Test User", 2023-10-12T18:48:00.750 INFO:teuthology.orchestra.run.smithi170.stdout: "email": "", 2023-10-12T18:48:00.751 INFO:teuthology.orchestra.run.smithi170.stdout: "suspended": 0, 2023-10-12T18:48:00.751 INFO:teuthology.orchestra.run.smithi170.stdout: "max_buckets": 1000, 2023-10-12T18:48:00.751 INFO:teuthology.orchestra.run.smithi170.stdout: "subusers": [], 2023-10-12T18:48:00.751 INFO:teuthology.orchestra.run.smithi170.stdout: "keys": [ 2023-10-12T18:48:00.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:00.752 INFO:teuthology.orchestra.run.smithi170.stdout: "user": "rgw-multisite-test-user", 2023-10-12T18:48:00.752 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "YZ80XOWP25Y5B60S2H7E", 2023-10-12T18:48:00.752 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "qB10ApcsrSFTujf0Ck8RTVEZlI6Xbt31oE7bVvXC" 2023-10-12T18:48:00.752 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:00.752 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T18:48:00.753 INFO:teuthology.orchestra.run.smithi170.stdout: "swift_keys": [], 2023-10-12T18:48:00.753 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": [], 2023-10-12T18:48:00.753 INFO:teuthology.orchestra.run.smithi170.stdout: "op_mask": "read, write, delete", 2023-10-12T18:48:00.753 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "", 2023-10-12T18:48:00.753 INFO:teuthology.orchestra.run.smithi170.stdout: "default_storage_class": "", 2023-10-12T18:48:00.754 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_tags": [], 2023-10-12T18:48:00.754 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_quota": { 2023-10-12T18:48:00.754 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T18:48:00.754 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T18:48:00.754 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T18:48:00.755 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T18:48:00.755 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T18:48:00.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.755 INFO:teuthology.orchestra.run.smithi170.stdout: "user_quota": { 2023-10-12T18:48:00.755 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T18:48:00.756 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T18:48:00.756 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T18:48:00.756 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T18:48:00.756 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T18:48:00.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.756 INFO:teuthology.orchestra.run.smithi170.stdout: "temp_url_keys": [], 2023-10-12T18:48:00.757 INFO:teuthology.orchestra.run.smithi170.stdout: "type": "rgw", 2023-10-12T18:48:00.757 INFO:teuthology.orchestra.run.smithi170.stdout: "mfa_ids": [] 2023-10-12T18:48:00.757 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:00.757 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T18:48:00.760 INFO:rgw_multi.tests:meta checkpoint 2023-10-12T18:48:00.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:00.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:00.762 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:00.802 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:00.803 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:00.836 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:48:00.847 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:48:00.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:00.848 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:48:00.848 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:48:00.745890Z" 2023-10-12T18:48:00.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:00.849 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:00.849 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:48:00.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:00.849 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:00.850 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:48:00.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.850 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:00.850 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:00.850 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:48:00.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:00.851 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:48:00.852 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:48:00.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:00.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:00.853 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:00.933 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:00.933 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:00.978 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:00.979 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:00.979 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:00.979 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:00.979 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:00.980 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:00.980 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:00.980 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.980 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:00.980 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:00.981 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:00.981 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:00.981 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:00.981 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:00.981 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:00.982 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:00.982 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:00.982 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:00.982 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:00.982 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:00.983 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.983 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:00.983 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:00.983 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:00.983 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:00.984 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:00.984 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:00.984 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:00.984 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:00.984 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:00.984 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:00.985 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:00.985 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.985 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:00.985 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:00.985 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:00.985 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:00.986 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:00.986 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:00.986 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:00.986 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:00.986 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:00.987 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:00.987 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:00.987 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.987 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:00.987 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:00.988 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:00.988 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:00.988 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:00.988 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:00.988 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:00.989 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:00.989 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:00.989 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:00.989 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:00.989 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:00.989 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:00.991 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:00.992 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:05.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:05.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:05.993 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:06.041 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:06.042 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:06.093 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:06.093 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:06.093 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:06.094 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:06.094 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:06.095 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:06.095 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:06.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:06.096 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:06.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:06.096 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:06.097 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:06.097 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:06.097 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:06.098 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:06.098 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:06.098 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:06.100 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:06.101 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:06.101 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:06.101 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:06.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:06.101 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:06.102 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:06.102 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:06.102 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:06.102 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:06.102 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:06.103 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:06.103 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:06.103 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:06.103 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:06.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:06.104 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:06.104 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:06.104 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:06.104 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:06.104 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:06.105 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:06.105 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:06.105 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:06.106 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:06.106 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:06.106 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:06.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:06.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:06.107 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:06.108 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:06.108 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:06.108 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:06.109 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:06.109 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:06.109 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:06.109 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:06.110 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:06.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:06.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:06.110 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:06.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:06.111 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:06.111 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:06.111 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:06.111 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:06.112 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:06.113 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:11.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:11.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:11.115 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:11.163 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:11.163 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:11.215 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:11.228 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:11.229 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:16.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:16.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:16.231 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:16.274 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:16.274 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:16.321 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:16.322 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:16.322 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:16.322 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:16.322 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:16.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:16.323 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:16.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:16.323 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:16.323 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:16.323 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:16.324 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:16.324 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:16.324 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:16.324 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:16.324 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:16.325 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:16.325 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:16.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:16.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:16.325 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:16.326 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:16.326 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:16.326 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:16.326 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:16.326 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:16.327 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:16.327 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:16.327 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:16.327 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:16.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:16.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:16.328 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:16.328 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:16.328 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:16.328 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:16.329 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:16.329 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:16.329 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:16.329 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:16.329 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:16.329 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:16.330 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:16.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:16.330 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:16.330 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:16.330 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:16.331 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:16.331 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:16.331 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:16.331 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:16.331 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:16.332 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:16.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:16.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:16.332 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:16.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:16.333 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:16.333 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:16.333 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:16.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:16.333 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:16.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:21.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:21.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:21.337 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:21.385 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:21.435 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:21.448 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:21.449 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:26.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:26.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:26.451 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:26.497 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:26.497 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:26.546 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:26.546 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:26.547 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:26.547 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:26.547 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:26.547 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:26.547 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:26.548 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:26.548 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:26.548 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:26.548 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:26.548 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:26.548 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:26.549 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:26.549 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:26.549 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:26.549 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:26.549 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:26.550 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:26.550 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:26.550 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:26.550 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:26.550 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:26.551 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:26.551 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:26.551 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:26.551 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:26.551 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:26.552 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:26.553 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:26.553 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:26.553 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:26.554 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:26.554 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:26.554 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:26.554 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:26.554 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:26.555 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:26.555 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:26.555 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:26.555 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:26.555 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:26.556 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:26.556 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:26.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:26.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:26.556 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:26.557 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:26.557 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:26.557 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:26.557 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:26.557 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:26.557 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:26.558 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:26.558 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:26.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:26.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:26.558 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:26.559 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:26.559 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:26.559 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:26.559 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:26.559 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:26.560 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:26.561 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:31.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:31.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:31.563 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:31.611 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:31.612 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:31.663 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:31.663 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:31.663 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:31.664 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:31.664 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:31.664 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:31.664 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:31.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:31.665 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:31.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:31.665 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:31.665 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:31.666 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:31.666 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:31.666 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:31.669 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:31.669 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:31.670 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:31.670 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:31.670 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:31.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:31.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:31.671 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:31.672 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:31.672 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:31.672 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:31.673 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:31.673 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:31.673 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:31.674 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:31.674 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:31.674 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:31.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:31.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:31.675 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:31.675 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:31.676 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:31.676 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:31.676 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:31.676 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:31.677 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:31.677 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:31.677 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:31.677 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:31.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:31.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:31.678 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:31.678 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:31.678 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:31.678 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:31.679 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:31.679 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:31.679 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:31.679 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:31.679 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:31.679 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:31.680 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:31.680 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:31.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:31.680 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:31.680 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:31.681 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:31.681 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:31.681 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:31.683 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:36.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:36.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:36.685 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:36.733 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:36.734 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:36.784 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:36.785 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:36.785 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:36.785 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:36.785 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:36.786 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:36.786 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:36.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:36.786 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:36.787 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:36.787 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:36.787 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:36.787 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:36.787 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:36.788 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:36.788 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:36.788 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:36.788 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:36.788 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:36.789 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:36.789 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:36.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:36.789 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:36.789 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:36.790 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:36.790 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:36.790 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:36.790 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:36.790 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:36.791 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:36.791 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:36.791 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:36.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:36.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:36.792 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:36.792 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:36.792 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:36.792 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:36.792 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:36.793 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:36.793 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:36.793 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:36.793 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:36.793 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:36.794 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:36.794 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:36.794 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:36.794 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:36.794 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:36.795 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:36.795 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:36.795 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:36.795 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:36.795 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:36.796 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:36.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:36.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:36.796 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:36.796 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:36.797 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:36.797 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:36.797 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:36.797 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:36.797 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:36.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:41.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:41.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:41.803 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:41.851 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:41.851 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:41.902 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:41.903 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:41.903 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:41.903 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:41.903 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:41.904 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:41.904 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:41.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:41.904 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:41.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:41.905 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:41.905 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:41.905 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:41.905 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:41.905 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:41.906 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:41.906 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:41.906 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:41.906 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:41.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:41.907 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:41.907 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:41.907 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:41.907 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:41.907 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:41.907 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:41.908 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:41.908 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:41.908 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:41.908 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:41.908 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:41.909 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:41.909 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:41.909 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:41.909 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:41.909 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:41.909 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:41.910 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:41.910 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:41.910 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:41.910 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:41.910 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:41.911 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:41.911 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:41.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:41.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:41.911 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:41.912 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:41.912 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:41.912 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:41.912 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:41.912 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:41.912 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:41.913 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:41.913 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:41.913 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:41.913 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:41.913 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:41.914 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:41.914 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:41.914 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:41.914 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:41.914 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:41.915 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:41.916 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:46.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:46.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:46.918 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:46.966 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:46.967 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:47.017 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:47.018 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:47.018 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:47.018 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:47.018 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:47.019 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:47.019 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:47.019 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:47.019 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:47.019 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:47.020 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:47.020 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:47.020 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:47.020 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:47.020 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:47.020 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:47.021 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:47.021 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:47.021 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:47.021 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:47.021 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:47.022 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:47.022 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:47.022 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:47.022 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:47.022 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:47.023 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:47.023 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:47.023 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:47.023 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:47.023 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:47.024 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:47.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:47.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:47.024 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:47.024 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:47.024 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:47.025 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:47.025 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:47.025 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:47.025 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:47.025 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:47.026 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:47.026 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:47.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:47.026 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:47.026 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:47.027 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:47.027 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:47.027 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:47.027 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:47.028 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:47.028 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:47.028 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:47.028 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:47.028 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:47.028 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:47.029 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:47.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:47.029 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:47.029 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:47.029 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:47.030 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:47.030 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:47.031 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:52.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:52.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:52.033 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:52.137 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:52.137 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:52.137 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:52.137 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:52.138 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:52.138 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:52.138 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:52.138 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:52.138 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:52.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:52.139 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:52.139 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:52.139 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:52.139 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:52.140 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:52.140 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:52.140 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:52.140 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:52.140 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:52.141 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:52.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:52.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:52.141 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:52.141 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:52.142 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:52.142 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:52.142 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:52.142 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:52.142 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:52.142 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:52.143 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:52.143 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:52.143 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:52.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:52.144 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:52.144 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:52.144 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:52.145 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:52.145 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:52.145 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:52.145 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:52.145 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:52.145 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:52.146 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:52.146 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:52.146 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:52.146 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:52.147 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:52.147 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:52.147 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:52.147 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:52.147 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:52.147 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:52.148 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:52.148 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:52.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:52.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:52.148 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:52.149 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:52.149 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:52.149 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:52.149 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:52.149 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:52.150 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:52.151 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:48:57.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:57.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:48:57.152 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:48:57.200 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:48:57.200 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:48:57.251 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:48:57.252 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:48:57.252 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:48:57.253 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:48:57.253 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:48:57.253 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:48:57.254 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:57.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:57.254 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:48:57.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:57.254 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:48:57.255 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:57.255 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:57.255 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:57.255 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:57.255 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:48:57.256 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:57.256 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:57.256 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:57.256 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:57.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:57.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:57.257 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:48:57.257 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:57.257 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:57.258 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:57.258 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:57.258 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:57.258 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:57.259 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:57.259 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:57.259 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:57.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:57.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:57.261 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:48:57.261 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:57.261 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:57.262 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:57.262 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:57.262 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:57.263 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:57.263 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:57.263 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:57.263 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:57.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:57.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:48:57.264 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:48:57.265 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:48:57.265 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:48:57.265 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:48:57.265 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:48:57.266 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:48:57.266 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:48:57.266 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:48:57.267 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:48:57.267 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:57.267 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:57.267 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:48:57.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:48:57.268 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:48:57.268 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:48:57.268 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:48:57.268 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:48:57.268 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:48:57.270 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:02.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:02.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:02.272 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:02.318 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:02.318 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:02.366 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:02.367 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:02.367 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:02.367 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:02.367 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:02.368 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:02.368 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:02.368 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:02.368 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:02.368 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:02.369 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:02.369 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:02.369 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:02.369 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:02.369 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:02.370 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:02.370 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:02.370 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:02.370 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:02.370 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:02.371 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:02.371 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:02.371 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:02.371 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:02.371 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:02.372 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:02.372 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:02.372 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:02.372 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:02.372 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:02.373 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:02.373 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:02.373 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:02.373 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:02.373 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:02.373 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:02.374 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:02.374 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:02.374 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:02.374 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:02.374 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:02.375 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:02.375 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:02.375 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:02.375 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:02.375 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:02.376 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:02.376 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:02.376 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:02.376 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:02.376 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:02.376 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:02.377 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:02.377 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:02.377 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:02.377 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:02.377 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:02.377 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:02.378 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:02.378 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:02.378 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:02.378 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:02.379 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:02.379 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:02.380 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:07.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:07.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:07.382 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:07.430 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:07.430 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:07.481 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:07.482 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:07.482 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:07.482 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:07.482 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:07.483 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:07.483 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:07.483 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:07.483 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:07.483 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:07.484 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:07.484 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:07.484 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:07.484 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:07.484 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:07.485 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:07.485 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:07.485 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:07.485 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:07.485 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:07.486 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:07.486 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:07.486 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:07.486 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:07.487 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:07.487 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:07.487 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:07.487 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:07.487 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:07.487 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:07.488 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:07.488 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:07.488 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:07.488 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:07.489 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:07.489 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:07.489 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:07.489 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:07.489 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:07.490 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:07.490 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:07.490 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:07.490 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:07.490 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:07.491 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:07.491 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:07.491 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:07.491 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:07.491 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:07.491 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:07.492 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:07.492 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:07.492 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:07.493 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:07.493 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:07.493 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:07.493 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:07.493 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:07.494 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:07.494 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:07.494 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:07.494 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:07.494 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:07.495 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:07.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:12.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:12.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:12.499 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:12.547 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:12.548 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:12.599 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:12.599 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:12.600 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:12.600 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:12.600 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:12.600 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:12.600 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:12.601 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:12.601 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:12.601 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:12.601 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:12.601 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:12.602 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:12.602 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:12.602 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:12.602 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:12.602 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:12.603 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:12.603 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:12.603 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:12.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:12.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:12.604 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:12.604 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:12.605 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:12.605 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:12.605 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:12.605 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:12.605 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:12.606 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:12.606 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:12.606 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:12.606 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:12.611 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:12.612 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:12.612 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:12.614 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:17.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:17.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:17.616 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:17.664 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:17.664 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:17.714 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:17.714 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:17.715 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:17.715 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:17.715 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:17.715 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:17.716 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:17.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:17.716 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:17.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:17.717 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:17.717 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:17.717 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:17.717 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:17.718 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:17.718 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:17.718 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:17.718 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:17.719 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:17.719 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:17.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:17.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:17.719 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:17.720 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:17.720 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:17.720 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:17.720 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:17.721 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:17.721 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:17.721 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:17.721 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:17.721 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:17.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:17.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:17.722 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:17.722 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:17.723 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:17.723 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:17.723 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:17.723 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:17.723 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:17.724 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:17.724 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:17.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:17.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:17.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:17.725 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:17.725 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:17.725 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:17.725 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:17.726 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:17.726 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:17.726 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:17.726 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:17.726 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:17.727 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:17.727 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:17.727 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:17.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:17.727 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:17.728 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:17.728 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:17.728 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:17.728 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:17.729 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:22.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:22.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:22.731 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:22.779 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:22.780 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:22.831 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:22.832 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:22.832 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:22.832 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:22.832 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:22.833 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:22.833 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:22.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:22.833 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:22.833 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:22.834 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:22.834 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:22.834 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:22.834 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:22.834 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:22.835 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:22.835 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:22.835 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:22.835 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:22.835 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:22.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:22.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:22.836 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:22.836 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:22.836 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:22.836 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:22.837 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:22.837 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:22.837 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:22.837 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:22.837 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:22.838 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:22.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:22.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:22.838 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:22.838 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:22.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:22.843 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:22.843 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:22.843 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:22.844 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:22.844 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:22.845 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:27.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:27.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:27.847 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:27.895 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:27.896 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:27.947 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:27.947 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:27.948 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:27.948 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:27.948 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:27.948 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:27.949 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:27.949 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:27.949 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:27.960 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:27.960 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:27.960 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:27.960 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:27.960 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:27.961 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:32.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:32.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:32.963 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:33.011 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:33.012 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:33.063 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:33.068 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:33.068 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:33.068 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:33.068 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:33.068 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:33.069 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:33.069 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:33.069 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:33.069 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:33.069 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:33.070 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:33.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:33.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:33.070 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:33.070 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:33.070 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:33.071 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:33.071 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:33.071 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:33.071 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:33.071 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:33.072 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:33.072 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:33.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:33.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:33.072 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:33.073 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:33.073 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:33.073 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:33.073 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:33.073 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:33.073 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:33.074 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:33.074 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:33.074 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:33.074 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:33.074 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:33.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:33.075 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:33.075 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:33.075 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:33.075 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:33.076 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:33.077 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:38.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:38.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:38.079 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:38.129 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:38.129 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:38.182 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:38.182 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:38.183 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:38.183 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:38.183 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:38.183 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:38.183 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:38.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:38.184 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:38.184 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:38.184 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:38.184 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:38.185 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:38.185 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:38.185 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:38.185 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:38.186 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:38.186 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:38.186 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:38.187 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:38.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:38.187 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:38.188 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:38.188 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:38.188 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:38.188 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:38.189 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:38.189 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:38.189 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:38.189 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:38.189 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:38.190 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:38.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:38.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:38.190 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:38.191 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:38.191 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:38.191 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:38.191 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:38.191 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:38.192 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:38.192 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:38.192 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:38.192 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:38.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:38.193 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:38.193 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:38.193 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:38.193 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:38.194 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:38.194 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:38.194 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:38.194 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:38.194 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:38.195 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:38.195 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:38.195 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:38.195 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:38.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:38.196 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:38.196 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:38.196 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:38.196 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:38.197 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:38.198 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:43.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:43.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:43.200 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:43.248 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:43.249 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:43.300 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:43.301 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:43.301 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:43.301 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:43.301 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:43.302 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:43.302 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:43.302 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:43.302 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:43.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:43.303 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:43.303 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:43.303 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:43.303 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:43.304 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:43.304 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:43.304 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:43.304 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:43.304 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:43.305 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:43.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:43.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:43.305 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:43.305 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:43.305 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:43.306 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:43.306 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:43.306 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:43.306 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:43.306 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:43.307 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:43.307 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:43.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:43.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:43.307 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:43.308 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:43.308 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:43.308 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:43.308 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:43.308 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:43.309 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:43.309 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:43.309 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:43.309 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:43.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:43.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:43.310 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:43.310 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:43.310 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:43.310 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:43.311 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:43.311 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:43.311 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:43.311 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:43.311 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:43.312 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:43.312 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:43.312 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:43.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:43.312 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:43.313 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:43.313 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:43.314 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:48.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:48.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:48.316 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:48.365 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:48.365 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:48.416 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:48.417 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:48.417 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:48.417 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:48.417 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:48.418 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:48.418 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:48.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:48.418 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:48.419 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:48.419 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:48.419 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:48.419 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:48.419 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:48.420 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:48.420 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:48.420 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:48.420 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:48.420 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:48.421 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:48.421 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:48.421 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:48.421 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:48.421 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:48.422 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:48.422 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:48.422 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:48.422 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:48.423 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:48.423 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:48.423 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:48.423 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:48.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:48.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:48.424 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:48.424 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:48.424 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:48.425 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:48.425 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:48.425 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:48.425 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:48.426 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:48.426 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:48.426 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:48.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:48.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:48.427 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:48.427 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:48.427 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:48.427 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:48.428 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:48.428 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:48.428 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:48.428 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:48.428 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:48.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:48.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:48.429 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:48.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:48.430 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:48.430 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:48.430 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:48.430 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:48.430 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:48.432 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:53.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:53.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:53.434 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:53.481 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:53.482 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:53.533 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:53.533 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:53.534 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:53.534 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:53.534 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:53.534 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:53.534 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:53.535 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:53.535 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:53.535 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:53.535 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:53.535 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:53.536 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:53.536 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:53.536 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:53.536 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:53.536 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:53.537 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:53.537 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:53.537 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:53.537 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:53.537 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:53.538 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:53.538 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:53.538 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:53.538 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:53.538 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:53.539 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:53.539 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:53.539 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:53.539 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:53.539 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:53.539 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:53.540 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:53.540 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:53.540 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:53.540 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:53.540 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:53.541 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:53.541 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:53.541 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:53.541 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:53.541 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:53.541 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:53.542 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:53.542 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:53.542 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:53.542 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:53.542 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:53.543 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:53.543 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:53.543 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:53.543 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:53.543 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:53.543 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:53.544 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:53.544 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:53.544 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:53.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:53.544 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:53.544 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:53.545 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:53.545 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:53.545 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:53.546 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136480.745890_22.1 target= 2023-10-12T18:49:58.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:58.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:58.548 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:58.595 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:58.596 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:58.648 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:58.648 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:58.648 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:58.649 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:58.649 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:58.649 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:58.649 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:58.649 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:58.650 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:58.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:58.650 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:58.650 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:58.651 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:58.651 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:49:58.651 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:58.651 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:58.651 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:58.652 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:49:58.652 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:58.652 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:58.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:58.653 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:58.653 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:58.653 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:58.653 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:58.653 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:58.654 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:58.654 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:58.654 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:58.654 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:58.655 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:58.655 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:58.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:58.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:58.655 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:58.656 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:58.656 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:58.656 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:58.656 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:58.656 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:58.657 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:58.657 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:58.657 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:58.657 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:58.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:58.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:58.658 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:58.658 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:58.659 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:58.659 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:58.659 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:58.659 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:58.659 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:58.660 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:58.660 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:58.660 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:58.660 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:58.660 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:58.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:58.661 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:58.661 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:58.661 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:58.661 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:58.662 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:58.663 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:49:58.663 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-12T18:49:58.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:58.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:58.665 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:58.713 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:58.713 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:58.782 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:49:58.783 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_status": { 2023-10-12T18:49:58.783 INFO:teuthology.orchestra.run.smithi187.stdout: "info": { 2023-10-12T18:49:58.783 INFO:teuthology.orchestra.run.smithi187.stdout: "status": "sync", 2023-10-12T18:49:58.783 INFO:teuthology.orchestra.run.smithi187.stdout: "num_shards": 4, 2023-10-12T18:49:58.783 INFO:teuthology.orchestra.run.smithi187.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:58.784 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.784 INFO:teuthology.orchestra.run.smithi187.stdout: "markers": [ 2023-10-12T18:49:58.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:58.784 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 0, 2023-10-12T18:49:58.785 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:58.785 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:58.785 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:49:58.785 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:58.785 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 1, 2023-10-12T18:49:58.786 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:58.786 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:49:58.786 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.786 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:58.787 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 1, 2023-10-12T18:49:58.787 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:58.787 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:58.787 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:58.788 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:58.788 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:58.788 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:58.788 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:58.788 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 2, 2023-10-12T18:49:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 3, 2023-10-12T18:49:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:58.792 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:58.792 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:58.792 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:58.792 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:58.792 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:49:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: "full_sync": { 2023-10-12T18:49:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: "total": 1, 2023-10-12T18:49:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: "complete": 1 2023-10-12T18:49:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.795 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:49:58.796 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-12T18:49:58.796 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-12T18:49:58.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:58.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:58.797 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:58.839 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:58.840 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:58.920 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:49:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_status": { 2023-10-12T18:49:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "info": { 2023-10-12T18:49:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "status": "sync", 2023-10-12T18:49:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "num_shards": 4, 2023-10-12T18:49:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "markers": [ 2023-10-12T18:49:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 0, 2023-10-12T18:49:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:49:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 1, 2023-10-12T18:49:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:49:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 1, 2023-10-12T18:49:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 2, 2023-10-12T18:49:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 3, 2023-10-12T18:49:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:49:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "full_sync": { 2023-10-12T18:49:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "total": 1, 2023-10-12T18:49:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "complete": 1 2023-10-12T18:49:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:58.932 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:49:58.933 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-12T18:49:58.933 INFO:rgw_multi.tests:meta checkpoint 2023-10-12T18:49:58.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:58.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:58.934 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:58.979 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:58.979 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:59.016 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:49:59.027 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:49:59.027 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:59.028 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:49:59.028 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:48:00.745890Z" 2023-10-12T18:49:59.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:59.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:59.028 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:59.029 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:49:59.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:59.029 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:59.029 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:59.029 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:49:59.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:59.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:59.030 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:59.030 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:49:59.030 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:59.030 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:49:59.031 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:49:59.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:59.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:59.033 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:59.074 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:59.074 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:59.120 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:49:59.121 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:49:59.121 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:49:59.121 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:49:59.121 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:49:59.121 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:59.122 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:59.122 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:59.122 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:49:59.122 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:59.122 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:49:59.123 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:59.123 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:59.123 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:49:59.123 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:59.123 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:49:59.124 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:59.124 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:49:59.124 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:59.124 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:59.124 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:59.124 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:59.125 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:49:59.125 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:59.125 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:59.125 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:59.125 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:59.126 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:59.126 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:59.126 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:59.126 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:59.126 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:59.127 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:59.127 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:59.127 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:49:59.127 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:59.127 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:59.127 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:59.128 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:59.128 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:59.128 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:59.128 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:59.128 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:59.129 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:59.129 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:59.129 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:49:59.129 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:49:59.129 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:49:59.129 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:49:59.130 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:49:59.130 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:49:59.130 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:49:59.130 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:49:59.130 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:49:59.131 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:49:59.131 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:59.131 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:59.131 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:49:59.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:49:59.132 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:49:59.132 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:49:59.132 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:49:59.132 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:49:59.132 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:49:59.133 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:49:59.134 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-12T18:49:59.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:59.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:59.135 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:59.177 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:59.177 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:59.235 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:49:59.236 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_status": { 2023-10-12T18:49:59.236 INFO:teuthology.orchestra.run.smithi187.stdout: "info": { 2023-10-12T18:49:59.236 INFO:teuthology.orchestra.run.smithi187.stdout: "status": "sync", 2023-10-12T18:49:59.236 INFO:teuthology.orchestra.run.smithi187.stdout: "num_shards": 4, 2023-10-12T18:49:59.237 INFO:teuthology.orchestra.run.smithi187.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:59.237 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.237 INFO:teuthology.orchestra.run.smithi187.stdout: "markers": [ 2023-10-12T18:49:59.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:59.238 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 0, 2023-10-12T18:49:59.238 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:59.238 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:59.239 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:49:59.239 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:59.239 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 1, 2023-10-12T18:49:59.239 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:59.240 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:49:59.240 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.240 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:59.241 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 1, 2023-10-12T18:49:59.241 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:59.241 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:59.241 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:59.241 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:59.242 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:59.242 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:59.242 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:59.242 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.242 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.243 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:59.243 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 2, 2023-10-12T18:49:59.243 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:59.243 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:59.244 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:59.244 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:59.244 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:59.244 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:59.244 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:59.244 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.245 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:59.245 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 3, 2023-10-12T18:49:59.245 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:59.246 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:59.246 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:59.246 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:59.246 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:59.246 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:59.246 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:59.247 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.247 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.247 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.247 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:49:59.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.248 INFO:teuthology.orchestra.run.smithi187.stdout: "full_sync": { 2023-10-12T18:49:59.248 INFO:teuthology.orchestra.run.smithi187.stdout: "total": 1, 2023-10-12T18:49:59.248 INFO:teuthology.orchestra.run.smithi187.stdout: "complete": 1 2023-10-12T18:49:59.248 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.248 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:49:59.250 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-12T18:49:59.250 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-12T18:49:59.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:59.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:49:59.251 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:49:59.331 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:49:59.332 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:49:59.405 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-10-12T18:49:59.405 INFO:teuthology.orchestra.run.smithi187.stdout: "sync_status": { 2023-10-12T18:49:59.406 INFO:teuthology.orchestra.run.smithi187.stdout: "info": { 2023-10-12T18:49:59.406 INFO:teuthology.orchestra.run.smithi187.stdout: "status": "sync", 2023-10-12T18:49:59.406 INFO:teuthology.orchestra.run.smithi187.stdout: "num_shards": 4, 2023-10-12T18:49:59.406 INFO:teuthology.orchestra.run.smithi187.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:49:59.406 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.407 INFO:teuthology.orchestra.run.smithi187.stdout: "markers": [ 2023-10-12T18:49:59.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:59.407 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 0, 2023-10-12T18:49:59.407 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:59.407 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:59.408 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:49:59.408 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:59.408 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 1, 2023-10-12T18:49:59.408 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:59.408 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:49:59.409 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.409 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:59.409 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 1, 2023-10-12T18:49:59.410 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:59.410 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:59.410 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:59.410 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:59.410 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:59.410 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:59.411 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:59.411 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.411 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:59.412 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 2, 2023-10-12T18:49:59.412 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:59.412 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:59.412 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:59.412 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:59.413 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:59.413 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:59.413 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:59.413 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.413 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2023-10-12T18:49:59.414 INFO:teuthology.orchestra.run.smithi187.stdout: "key": 3, 2023-10-12T18:49:59.414 INFO:teuthology.orchestra.run.smithi187.stdout: "val": { 2023-10-12T18:49:59.414 INFO:teuthology.orchestra.run.smithi187.stdout: "state": 1, 2023-10-12T18:49:59.414 INFO:teuthology.orchestra.run.smithi187.stdout: "marker": "", 2023-10-12T18:49:59.415 INFO:teuthology.orchestra.run.smithi187.stdout: "next_step_marker": "", 2023-10-12T18:49:59.415 INFO:teuthology.orchestra.run.smithi187.stdout: "total_entries": 0, 2023-10-12T18:49:59.415 INFO:teuthology.orchestra.run.smithi187.stdout: "pos": 0, 2023-10-12T18:49:59.415 INFO:teuthology.orchestra.run.smithi187.stdout: "timestamp": "0.000000", 2023-10-12T18:49:59.415 INFO:teuthology.orchestra.run.smithi187.stdout: "realm_epoch": 2 2023-10-12T18:49:59.415 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.416 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.416 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-10-12T18:49:59.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2023-10-12T18:49:59.416 INFO:teuthology.orchestra.run.smithi187.stdout: "full_sync": { 2023-10-12T18:49:59.416 INFO:teuthology.orchestra.run.smithi187.stdout: "total": 1, 2023-10-12T18:49:59.417 INFO:teuthology.orchestra.run.smithi187.stdout: "complete": 1 2023-10-12T18:49:59.417 INFO:teuthology.orchestra.run.smithi187.stdout: } 2023-10-12T18:49:59.417 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-10-12T18:49:59.418 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-12T18:49:59.418 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2023-10-12T18:49:59.424 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-1 2023-10-12T18:50:02.166 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-2 2023-10-12T18:50:05.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:05.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:05.184 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:05.242 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:05.242 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:05.276 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:50:05.288 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:50:05.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:05.288 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:05.289 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:48:00.745890Z" 2023-10-12T18:50:05.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:05.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:05.289 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136602.182231_67.1", 2023-10-12T18:50:05.289 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:02.182231Z" 2023-10-12T18:50:05.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:05.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:05.290 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:05.290 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:05.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:05.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:05.291 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:05.291 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:05.291 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:05.292 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:50:05.293 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:50:05.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:05.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:05.294 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:05.336 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:05.336 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:05.382 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:05.383 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:05.383 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:05.383 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:05.383 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:05.383 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:05.384 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:05.384 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:05.384 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:05.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:05.384 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:05.384 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:05.385 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:05.385 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:05.386 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:05.386 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:05.386 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:05.386 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:05.387 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:05.387 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:05.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:05.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:05.387 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:05.388 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:05.388 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:05.388 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136602.151412_60.1", 2023-10-12T18:50:05.388 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:05.389 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:05.389 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:05.389 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:02.151412Z", 2023-10-12T18:50:05.389 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:05.390 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:05.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:05.391 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:05.391 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:05.391 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:05.391 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:05.391 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:05.392 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:05.392 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:05.392 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:05.392 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:05.392 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:05.393 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:05.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:05.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:05.393 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:05.393 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:05.394 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:05.394 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:05.394 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:05.394 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:05.394 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:05.395 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:05.395 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:05.395 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:05.395 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:05.396 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:05.396 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:05.396 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:05.396 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:05.397 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:05.397 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:05.397 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:05.398 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697136602.182231_67.1 target=1_1697136602.151412_60.1 2023-10-12T18:50:10.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:10.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:10.401 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:10.449 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:10.449 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:10.500 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:10.501 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:10.501 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:10.501 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:10.501 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:10.502 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:10.502 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.502 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:10.503 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.503 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:10.503 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:10.503 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:10.503 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:10.504 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:10.504 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:10.504 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:10.504 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:10.504 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.505 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.505 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.505 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:10.505 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:10.506 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:10.506 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136602.182231_67.1", 2023-10-12T18:50:10.506 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:10.506 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:10.506 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:10.507 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:02.182231Z", 2023-10-12T18:50:10.507 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.507 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.507 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.508 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:10.508 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:10.508 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:10.508 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:10.509 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:10.509 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:10.509 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:10.509 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:10.510 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.510 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.511 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.511 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:10.511 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:10.511 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:10.511 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:10.512 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:10.512 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:10.512 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:10.512 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:10.512 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.513 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.513 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.514 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:10.514 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.515 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:10.515 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:10.515 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:10.516 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.516 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:10.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:50:10.527 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2023-10-12T18:50:10.528 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-3 2023-10-12T18:50:10.542 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-4 2023-10-12T18:50:10.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:10.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:10.566 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:10.607 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:10.608 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:10.642 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:50:10.654 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:50:10.654 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.654 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:10.654 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:48:00.745890Z" 2023-10-12T18:50:10.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.655 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136610.555508_75.1", 2023-10-12T18:50:10.655 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:10.555508Z" 2023-10-12T18:50:10.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.656 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.656 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:10.656 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:10.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.657 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:10.657 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:10.657 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.658 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:50:10.659 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:50:10.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:10.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:10.661 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:10.742 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:10.743 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:10.789 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:10.789 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:10.789 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:10.790 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:10.790 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:10.790 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:10.791 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.791 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:10.792 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.792 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:10.792 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:10.793 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:10.793 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:10.793 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:10.794 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:10.794 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:10.794 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:10.795 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.795 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.795 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.796 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.796 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:10.796 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:10.797 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:10.797 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136610.534293_68.1", 2023-10-12T18:50:10.797 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:10.797 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:10.798 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:10.798 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:10.534293Z", 2023-10-12T18:50:10.798 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.798 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.798 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.799 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.799 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:10.799 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:10.799 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:10.799 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:10.800 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:10.800 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:10.800 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:10.800 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:10.800 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.801 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.801 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.801 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:10.801 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:10.801 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:10.802 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:10.802 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:10.802 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:10.802 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:10.802 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:10.804 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:10.806 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697136610.555508_75.1 target=1_1697136610.534293_68.1 2023-10-12T18:50:15.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:15.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:15.808 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:15.855 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:15.855 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:15.906 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:15.907 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:15.907 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:15.907 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:15.907 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:15.907 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:15.908 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:15.908 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:15.908 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:15.908 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:15.908 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:15.909 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:15.909 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:15.909 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:15.909 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:15.909 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:15.910 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:15.910 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:15.910 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:15.910 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:15.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:15.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:15.911 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:15.911 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:15.911 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:15.912 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136610.555508_75.1", 2023-10-12T18:50:15.912 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:15.912 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:15.912 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:15.912 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:10.555508Z", 2023-10-12T18:50:15.912 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:15.913 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:15.913 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:15.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:15.913 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:15.913 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:15.914 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:15.914 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:15.914 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:15.914 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:15.914 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:15.915 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:15.915 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:15.915 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:15.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:15.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:15.915 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:15.916 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:15.916 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:15.916 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:15.916 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:15.916 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:15.917 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:15.917 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:15.917 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:15.917 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:15.917 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:15.917 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:15.918 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:15.918 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:15.918 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:15.918 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:15.918 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:15.919 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:15.920 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:50:15.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:15.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:15.991 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:16.032 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:16.033 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:16.066 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:50:16.077 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:50:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:48:00.745890Z" 2023-10-12T18:50:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136610.555508_75.1", 2023-10-12T18:50:16.079 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:10.555508Z" 2023-10-12T18:50:16.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.079 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.079 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:16.079 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:16.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.080 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:50:16.082 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:50:16.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:16.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:16.083 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:16.163 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:16.163 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:16.210 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:16.210 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:16.214 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:16.214 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.214 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136610.555508_75.1", 2023-10-12T18:50:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:10.555508Z", 2023-10-12T18:50:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.217 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.217 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.220 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:16.220 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:16.220 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:16.220 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:16.221 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:16.221 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:16.221 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:16.221 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:16.221 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.222 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.222 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.222 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:16.222 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.222 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:16.223 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:16.223 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:16.223 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.223 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:16.224 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:50:16.233 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2023-10-12T18:50:16.234 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-5 2023-10-12T18:50:16.247 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-6 2023-10-12T18:50:16.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:16.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:16.270 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:16.310 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:16.311 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:16.344 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:50:16.356 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:50:16.357 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.357 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:16.357 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:48:00.745890Z" 2023-10-12T18:50:16.357 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136616.258652_83.1", 2023-10-12T18:50:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:16.258652Z" 2023-10-12T18:50:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.359 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:16.359 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:16.359 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.359 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.359 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:16.360 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:16.360 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.360 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:50:16.361 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:50:16.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:16.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:16.363 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:16.443 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:16.443 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:16.489 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:16.489 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:16.490 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:16.490 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:16.490 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:16.490 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:16.490 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.491 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.491 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:16.491 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.491 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:16.491 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:16.492 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:16.492 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:16.492 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:16.492 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:16.492 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:16.493 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:16.493 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.493 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.493 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.493 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.494 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:16.494 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:16.494 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:16.494 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136616.239423_76.1", 2023-10-12T18:50:16.494 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:16.495 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:16.495 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:16.495 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:16.239423Z", 2023-10-12T18:50:16.495 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.495 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.496 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.496 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.496 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:16.496 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:16.496 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:16.497 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:16.497 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:16.497 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:16.497 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:16.497 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:16.498 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.498 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.498 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.498 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:16.499 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:16.499 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:16.499 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:16.499 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:16.499 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:16.500 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:16.500 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:16.500 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:16.500 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:16.500 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.501 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:16.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:16.501 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:16.501 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:16.502 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:16.502 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:16.502 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:16.503 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697136616.258652_83.1 target=1_1697136616.239423_76.1 2023-10-12T18:50:21.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:21.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:21.505 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:21.553 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:21.553 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:21.605 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:21.605 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:21.606 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:21.606 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:21.606 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:21.606 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:21.606 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136616.258652_83.1", 2023-10-12T18:50:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:16.258652Z", 2023-10-12T18:50:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.617 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:21.619 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:50:21.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:21.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:21.709 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:21.749 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:21.750 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:21.784 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:50:21.795 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:50:21.795 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.796 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:21.796 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:48:00.745890Z" 2023-10-12T18:50:21.796 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.796 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.797 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136621.683853_91.1", 2023-10-12T18:50:21.797 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:21.683853Z" 2023-10-12T18:50:21.797 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.797 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.797 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:21.798 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:21.798 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.798 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.798 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:21.799 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:21.799 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.799 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:50:21.800 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:50:21.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:21.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:21.801 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:21.882 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:21.883 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:21.929 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:21.929 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:21.929 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:21.930 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:21.930 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:21.930 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:21.930 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.931 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.931 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:21.931 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.931 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:21.931 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:21.932 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:21.932 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:21.932 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:21.932 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:21.933 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:21.933 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:21.933 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.933 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.933 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.934 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.934 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:21.934 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:21.934 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:21.934 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136621.638418_84.1", 2023-10-12T18:50:21.935 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:21.935 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:21.935 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:21.935 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:21.638418Z", 2023-10-12T18:50:21.935 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.936 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.936 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.936 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.936 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:21.936 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:21.937 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:21.937 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:21.937 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:21.937 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:21.937 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:21.938 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:21.938 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.938 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.938 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.938 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:21.939 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:21.939 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:21.939 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:21.939 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:21.939 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:21.939 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:21.940 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:21.940 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:21.940 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:21.940 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.940 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.941 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:21.941 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:21.941 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:21.941 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:21.942 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:21.942 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:21.942 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:21.943 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697136621.683853_91.1 target=1_1697136621.638418_84.1 2023-10-12T18:50:26.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:26.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:26.945 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:26.995 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:26.995 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:27.046 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:27.046 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:27.046 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:27.046 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:27.047 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:27.047 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:27.047 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.047 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:27.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.048 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:27.048 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:27.048 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:27.048 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:27.049 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:27.049 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:27.049 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:27.049 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:27.049 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.049 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.050 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:27.050 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:27.050 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:27.051 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136621.683853_91.1", 2023-10-12T18:50:27.051 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:27.051 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:27.051 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:27.051 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:21.683853Z", 2023-10-12T18:50:27.052 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.052 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.052 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:27.053 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:27.053 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:27.053 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:27.053 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:27.053 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:27.053 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:27.054 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:27.054 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.054 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.055 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:27.055 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:27.055 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:27.055 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:27.055 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:27.056 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:27.056 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:27.056 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:27.056 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.057 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:27.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.057 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:27.057 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:27.057 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:27.058 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.058 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:27.059 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:50:27.064 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2023-10-12T18:50:27.065 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-7 2023-10-12T18:50:27.077 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-8 2023-10-12T18:50:27.271 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T18:50:27.269+0000 7f6645676700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T18:50:27.271 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T18:50:27.270+0000 7f6648e7d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T18:50:27.271 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T18:50:27.270+0000 7f6643672700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T18:50:27.271 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T18:50:27.270+0000 7f6647e7b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T18:50:27.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:27.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:27.332 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:27.373 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:27.373 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:27.410 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:50:27.422 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:50:27.422 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.422 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:27.423 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:48:00.745890Z" 2023-10-12T18:50:27.423 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.423 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.423 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136627.090522_99.1", 2023-10-12T18:50:27.423 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:27.090522Z" 2023-10-12T18:50:27.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.424 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:27.424 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:27.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.425 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:27.425 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:27.425 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.425 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:50:27.426 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:50:27.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:27.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:27.428 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:27.509 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:27.509 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:27.560 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:27.561 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:27.561 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:27.561 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:27.561 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:27.561 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:27.562 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.562 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.562 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:27.562 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.562 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:27.563 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:27.563 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:27.563 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:27.563 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:27.564 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:27.564 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:27.564 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:27.564 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.564 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.564 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.565 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.565 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:27.565 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:27.565 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:27.565 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136627.070176_92.1", 2023-10-12T18:50:27.566 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:27.566 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:27.566 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:27.566 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:27.070176Z", 2023-10-12T18:50:27.566 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.567 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.567 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.567 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:27.567 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:27.568 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:27.568 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:27.568 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:27.568 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:27.568 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:27.569 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:27.569 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.569 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:27.569 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:27.570 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:27.570 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:27.570 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:27.570 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:27.570 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:27.571 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:27.571 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:27.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:27.571 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.572 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.572 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:27.572 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:27.572 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:27.572 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:27.572 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:27.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:27.573 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:27.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697136627.090522_99.1 target=1_1697136627.070176_92.1 2023-10-12T18:50:32.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:32.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:32.576 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:32.620 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:32.620 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:32.668 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:32.668 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:32.668 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:32.668 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:32.669 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:32.669 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:32.669 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:32.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:32.670 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:32.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:32.670 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:32.670 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:32.670 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:32.671 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136480.745890_22.1", 2023-10-12T18:50:32.671 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:32.671 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:32.672 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:32.672 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:48:00.745890Z", 2023-10-12T18:50:32.672 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:32.672 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:32.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:32.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:32.673 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:32.673 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:32.673 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:32.673 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136627.090522_99.1", 2023-10-12T18:50:32.674 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:32.674 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:32.674 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:32.674 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:27.090522Z", 2023-10-12T18:50:32.674 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:32.675 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:32.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:32.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:32.675 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:32.675 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:32.676 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:32.676 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:32.676 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:32.677 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:32.677 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:32.677 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:32.677 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:32.677 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:32.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:32.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:32.678 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:32.678 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:32.678 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:32.679 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:32.679 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:32.679 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:32.679 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:32.679 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:32.679 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:32.680 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:32.680 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:32.681 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:32.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:32.681 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:32.681 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:32.681 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:32.682 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:32.682 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:32.683 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:50:32.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:32.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:32.685 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:50:32.733 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:32.733 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:32.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:32.773+0000 7f5614ada500 1 bucket sync caught up with source: 2023-10-12T18:50:32.774 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, 00000000001.3.6, , 00000000001.5.6, , , , 00000000001.7.6, 00000000001.2.6, , ] 2023-10-12T18:50:32.774 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, 00000000001.3.6, , 00000000001.5.6, , , , 00000000001.7.6, 00000000001.2.6, , ] 2023-10-12T18:50:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:32.773+0000 7f5614ada500 0 bucket checkpoint complete 2023-10-12T18:50:32.786 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-7 zones={a2, a1} 2023-10-12T18:50:32.791 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:50:32.795 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-12T18:50:32.796 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-12T18:50:32.796 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-12T18:50:32.796 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T18:50:32.797 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:50:32.800 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-12T18:50:32.801 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-12T18:50:32.801 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-12T18:50:32.802 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T18:50:32.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T18:50:32.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T18:50:32.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T18:50:32.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T18:50:32.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T18:50:32.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T18:50:32.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T18:50:33.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T18:50:33.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T18:50:33.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:33.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:33.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:33.134 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-7 zones={a2, a1} 2023-10-12T18:50:33.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:33.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:33.136 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:50:33.185 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:33.185 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:33.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:33.224+0000 7f632cf94500 1 bucket sync caught up with source: 2023-10-12T18:50:33.226 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, 00000000001.64.6, , 00000000001.3.6, , , , 00000000001.13.6, 00000000001.44.6, , ] 2023-10-12T18:50:33.226 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, 00000000001.64.6, , 00000000001.3.6, , , , 00000000001.13.6, 00000000001.44.6, , ] 2023-10-12T18:50:33.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:33.224+0000 7f632cf94500 0 bucket checkpoint complete 2023-10-12T18:50:33.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-8 zones={a1, a2} 2023-10-12T18:50:33.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:50:33.247 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-12T18:50:33.247 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-12T18:50:33.248 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-12T18:50:33.248 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T18:50:33.249 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:50:33.252 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-12T18:50:33.252 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-12T18:50:33.252 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-12T18:50:33.253 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T18:50:33.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T18:50:33.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T18:50:33.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T18:50:33.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T18:50:33.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T18:50:33.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T18:50:33.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T18:50:33.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T18:50:33.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T18:50:33.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:33.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:33.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:33.533 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-8 zones={a1, a2} 2023-10-12T18:50:33.534 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2023-10-12T18:50:33.534 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-9 2023-10-12T18:50:33.547 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-10 2023-10-12T18:50:33.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:33.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:33.630 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:33.671 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:33.672 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:33.672 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T18:50:33.671+0000 7f663c664700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T18:50:33.708 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:50:33.720 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:50:33.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:33.720 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.559518_32.1", 2023-10-12T18:50:33.721 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.559518Z" 2023-10-12T18:50:33.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:33.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:33.721 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:50:33.721 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:50:33.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:33.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:33.722 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:33.722 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:33.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:33.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:33.723 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:33.723 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:33.723 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:33.723 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:50:33.724 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:50:33.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:33.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:33.726 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:33.807 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:33.808 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:33.855 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:33.855 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:33.855 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:33.856 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:33.856 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:33.856 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:33.857 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:33.857 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:33.857 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:33.858 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:33.858 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:33.858 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:33.859 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:33.859 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.554052_29.1", 2023-10-12T18:50:33.859 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:33.860 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:33.860 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:33.860 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.554052Z", 2023-10-12T18:50:33.860 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:33.861 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:33.861 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:33.861 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:33.861 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:33.861 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:33.861 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:33.862 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.539659_100.1", 2023-10-12T18:50:33.862 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:33.862 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:33.862 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:33.863 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.539659Z", 2023-10-12T18:50:33.863 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:33.863 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:33.863 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:33.863 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:33.863 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:33.864 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:33.864 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:33.864 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:33.864 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:33.864 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:33.865 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:33.865 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:33.865 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:33.865 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:33.865 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:33.866 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:33.866 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:33.866 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:33.866 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:33.866 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:33.867 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:33.867 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:33.867 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:33.867 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:33.867 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:33.868 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:33.868 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:33.868 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:33.868 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:33.869 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:33.869 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:33.869 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:33.869 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:33.870 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:33.871 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136633.559518_32.1 target=1_1697136633.554052_29.1, shard=1 master=1_1697136633.545339_103.1 target=1_1697136633.539659_100.1 2023-10-12T18:50:38.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:38.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:38.873 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:38.921 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:38.921 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:38.972 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:38.973 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:38.973 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:38.973 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:38.973 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:38.974 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:38.974 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:38.974 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:38.974 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:38.974 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.559518_32.1", 2023-10-12T18:50:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:38.976 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:38.976 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.559518Z", 2023-10-12T18:50:38.976 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:38.976 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:38.976 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:38.977 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:38.977 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:38.977 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:38.977 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:38.977 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.539659_100.1", 2023-10-12T18:50:38.978 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:38.978 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:38.978 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:38.978 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.539659Z", 2023-10-12T18:50:38.978 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:38.978 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:38.979 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:38.979 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:38.979 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:38.979 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:38.979 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:38.980 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:38.980 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:38.980 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:38.980 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:38.980 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:38.981 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:38.981 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:38.981 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:38.981 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:38.981 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:38.981 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:38.982 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:38.982 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:38.982 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:38.982 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:38.982 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:38.983 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:38.983 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:38.983 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:38.983 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:38.983 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:38.983 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:38.984 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:38.984 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:38.984 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:38.984 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:38.984 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:38.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697136633.545339_103.1 target=1_1697136633.539659_100.1 2023-10-12T18:50:43.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:43.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:43.988 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:44.037 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:44.037 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:44.089 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:44.089 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:44.089 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:44.090 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:44.090 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:44.090 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:44.090 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:44.090 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:44.091 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:44.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:44.091 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:44.091 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:44.091 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:44.092 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.559518_32.1", 2023-10-12T18:50:44.092 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:44.092 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:44.092 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:44.092 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.559518Z", 2023-10-12T18:50:44.093 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:44.093 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:44.093 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:44.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:44.094 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:44.094 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:44.095 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:44.095 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:50:44.095 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:44.095 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:44.095 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:44.096 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:50:44.096 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:44.096 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:44.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:44.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:44.098 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:44.098 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:44.098 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:44.098 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:44.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:44.099 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:44.099 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:44.099 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:44.099 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:44.099 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:44.100 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:44.100 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:44.100 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:44.100 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:44.100 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:44.101 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:44.101 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:44.101 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:44.101 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:44.101 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:44.102 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:44.102 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:44.102 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:44.102 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:44.103 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:50:44.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:44.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:44.105 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:50:44.156 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:44.156 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:44.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:44.199+0000 7fb148709500 1 bucket sync caught up with source: 2023-10-12T18:50:44.200 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-10-12T18:50:44.200 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, 00000000001.7.6, , , , , , , , , ] 2023-10-12T18:50:44.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:44.199+0000 7fb148709500 0 bucket checkpoint complete 2023-10-12T18:50:44.212 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-9 zones={a2, a1} 2023-10-12T18:50:44.217 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:50:44.266 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T18:50:44.267 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:50:44.270 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T18:50:44.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:44.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:44.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:44.344 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-9 zones={a2, a1} 2023-10-12T18:50:44.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:44.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:44.346 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:50:44.394 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:44.394 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:44.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:44.435+0000 7f8e2c2db500 1 bucket sync caught up with source: 2023-10-12T18:50:44.436 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, 00000000001.87.6, , , , , , , , , ] 2023-10-12T18:50:44.436 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, 00000000001.87.6, , , , , , , , , ] 2023-10-12T18:50:44.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:44.435+0000 7f8e2c2db500 0 bucket checkpoint complete 2023-10-12T18:50:44.448 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-10 zones={a1, a2} 2023-10-12T18:50:44.453 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:50:44.455 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T18:50:44.456 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:50:44.459 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T18:50:44.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:44.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:44.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T18:50:44.577 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-10 zones={a1, a2} 2023-10-12T18:50:44.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:44.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:44.586 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:50:44.636 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:44.636 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:44.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:44.676+0000 7fc547e41500 1 waiting for incremental sync to catch up: 2023-10-12T18:50:44.677 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-10-12T18:50:44.677 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, 00000000002.9.6, , , , , , , , , ] 2023-10-12T18:50:49.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:49.677+0000 7fc547e41500 1 bucket sync caught up with source: 2023-10-12T18:50:49.678 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, 00000000002.9.6, , , , , , , , , ] 2023-10-12T18:50:49.678 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, 00000000002.9.6, , , , , , , , , ] 2023-10-12T18:50:49.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:49.677+0000 7fc547e41500 0 bucket checkpoint complete 2023-10-12T18:50:49.693 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-9 zones={a2, a1} 2023-10-12T18:50:49.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:50:49.702 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:50:49.710 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-9 zones={a2, a1} 2023-10-12T18:50:49.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:49.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:50:49.719 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:50:49.770 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:49.771 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:49.813 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:49.812+0000 7f0f2ff28500 1 waiting for incremental sync to catch up: 2023-10-12T18:50:49.813 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, 00000000001.87.6, , , , , , , , , ] 2023-10-12T18:50:49.813 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, 00000000002.139.6, , , , , , , , , ] 2023-10-12T18:50:54.815 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:54.813+0000 7f0f2ff28500 1 bucket sync caught up with source: 2023-10-12T18:50:54.815 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, 00000000002.139.6, , , , , , , , , ] 2023-10-12T18:50:54.815 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, 00000000002.139.6, , , , , , , , , ] 2023-10-12T18:50:54.815 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:50:54.813+0000 7f0f2ff28500 0 bucket checkpoint complete 2023-10-12T18:50:54.829 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-10 zones={a1, a2} 2023-10-12T18:50:54.835 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:50:54.837 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:50:54.892 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-10 zones={a1, a2} 2023-10-12T18:50:54.892 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2023-10-12T18:50:54.893 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-11 2023-10-12T18:50:54.907 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-12 2023-10-12T18:50:54.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:54.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:54.946 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:54.988 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:54.989 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:55.023 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:50:55.035 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:50:55.035 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:55.035 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136654.942190_63.1", 2023-10-12T18:50:55.036 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:54.942190Z" 2023-10-12T18:50:55.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:55.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:55.036 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:50:55.037 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:50:55.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:55.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:55.037 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:55.037 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:55.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:55.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:55.038 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:55.038 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:50:55.039 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:55.039 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:50:55.040 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:50:55.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:55.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:50:55.041 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:50:55.122 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:50:55.123 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:50:55.169 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:50:55.170 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:50:55.170 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:50:55.170 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:50:55.170 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:50:55.170 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:50:55.171 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:55.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:55.171 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:50:55.171 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:55.171 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:50:55.172 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:55.172 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:55.172 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136654.898969_52.1", 2023-10-12T18:50:55.172 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:55.172 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:50:55.173 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:55.173 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:54.898969Z", 2023-10-12T18:50:55.173 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:55.173 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:55.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:55.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:55.174 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:50:55.174 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:55.174 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:55.174 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:50:55.174 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:55.175 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:55.175 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:55.175 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:50:55.175 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:55.175 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:55.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:55.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:55.176 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:50:55.176 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:55.176 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:55.177 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:55.177 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:55.177 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:55.177 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:55.177 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:55.178 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:55.178 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:55.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:55.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:50:55.178 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:50:55.179 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:50:55.179 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:50:55.179 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:50:55.179 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:50:55.179 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:50:55.179 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:50:55.180 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:50:55.180 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:50:55.180 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:55.180 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:55.180 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:50:55.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:50:55.181 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:50:55.181 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:50:55.181 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:50:55.181 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:50:55.181 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:50:55.183 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136654.942190_63.1 target=1_1697136654.898969_52.1 2023-10-12T18:51:00.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:00.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:00.186 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:00.232 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:00.233 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:00.281 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:00.282 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:00.282 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:00.282 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:00.282 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:00.283 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:51:00.283 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:00.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:00.283 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:00.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:00.284 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:00.284 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:00.284 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:00.285 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136654.942190_63.1", 2023-10-12T18:51:00.285 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:00.285 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:51:00.285 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:00.285 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:54.942190Z", 2023-10-12T18:51:00.286 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:00.286 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:00.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:00.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:00.287 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:00.287 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:00.288 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:00.288 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:00.288 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:00.289 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:00.289 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:00.289 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:51:00.289 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:00.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:00.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:00.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:00.291 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:00.291 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:00.291 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:00.292 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:00.292 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:00.292 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:00.292 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:00.293 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:00.293 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:00.293 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:00.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:00.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:00.294 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:00.294 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:00.294 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:00.295 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:00.295 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:00.295 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:00.296 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:00.296 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:00.296 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:00.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:00.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:00.297 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:00.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:00.297 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:00.297 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:51:00.297 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:51:00.298 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:00.298 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:00.299 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:51:00.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:00.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:00.313 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:00.362 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:00.362 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:00.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:00.402+0000 7fe942e10500 1 waiting for incremental sync to catch up: 2023-10-12T18:51:00.403 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , , , ] 2023-10-12T18:51:00.403 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.6.11, , , , , ] 2023-10-12T18:51:05.404 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:05.403+0000 7fe942e10500 1 bucket sync caught up with source: 2023-10-12T18:51:05.404 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.8.10, , , , , ] 2023-10-12T18:51:05.404 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.6.11, , , , , ] 2023-10-12T18:51:05.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:05.403+0000 7fe942e10500 0 bucket checkpoint complete 2023-10-12T18:51:05.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:05.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:05.435 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:05.487 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:05.487 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:05.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:05.529+0000 7faca6a78500 1 waiting for incremental sync to catch up: 2023-10-12T18:51:05.530 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , , , ] 2023-10-12T18:51:05.530 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.74.11, , , , , ] 2023-10-12T18:51:10.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:10.530+0000 7faca6a78500 1 bucket sync caught up with source: 2023-10-12T18:51:10.532 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.76.10, , , , , ] 2023-10-12T18:51:10.532 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.74.11, , , , , ] 2023-10-12T18:51:10.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:10.530+0000 7faca6a78500 0 bucket checkpoint complete 2023-10-12T18:51:10.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:10.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:10.892 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:10.943 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:10.943 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:10.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:10.990+0000 7fcbcd9d7500 1 bucket sync caught up with source: 2023-10-12T18:51:10.991 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000002.7.11, 00000000010.22.6, ] 2023-10-12T18:51:10.992 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000002.7.11, 00000000010.22.6, ] 2023-10-12T18:51:10.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:10.990+0000 7fcbcd9d7500 0 bucket checkpoint complete 2023-10-12T18:51:11.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:11.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:11.005 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:11.055 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:11.055 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:11.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:11.097+0000 7f35ed7b9500 1 bucket sync caught up with source: 2023-10-12T18:51:11.099 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000002.95.11, , , 00000000010.130.6, 00000000002.113.11, ] 2023-10-12T18:51:11.099 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.95.11, , , 00000000010.130.6, 00000000002.113.11, ] 2023-10-12T18:51:11.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:11.098+0000 7f35ed7b9500 0 bucket checkpoint complete 2023-10-12T18:51:11.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-11 zones={a2, a1} 2023-10-12T18:51:11.115 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:51:11.119 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:11.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T18:51:11.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T18:51:11.120 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:51:11.123 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:11.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T18:51:11.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T18:51:11.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:11.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:11.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:11.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:11.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:11.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:11.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:11.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:11.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:11.423 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-11 zones={a2, a1} 2023-10-12T18:51:11.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:11.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:11.424 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:11.472 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:11.473 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:11.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:11.512+0000 7f9651995500 1 bucket sync caught up with source: 2023-10-12T18:51:11.513 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000002.4.11, , , 00000000002.13.11, 00000000010.24.6, ] 2023-10-12T18:51:11.514 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.4.11, , , 00000000002.13.11, 00000000010.24.6, ] 2023-10-12T18:51:11.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:11.512+0000 7f9651995500 0 bucket checkpoint complete 2023-10-12T18:51:11.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:11.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:11.526 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:11.574 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:11.575 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:11.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:11.614+0000 7fbb1c43e500 1 bucket sync caught up with source: 2023-10-12T18:51:11.615 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.76.10, , , 00000000010.257.6, 00000000002.196.11, ] 2023-10-12T18:51:11.616 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000003.76.10, , , 00000000010.257.6, 00000000002.196.11, ] 2023-10-12T18:51:11.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:11.614+0000 7fbb1c43e500 0 bucket checkpoint complete 2023-10-12T18:51:11.627 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-12 zones={a2, a1} 2023-10-12T18:51:11.632 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:51:11.635 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:11.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T18:51:11.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T18:51:11.636 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:51:11.638 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:11.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T18:51:11.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T18:51:11.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:11.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:11.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:11.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:11.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:11.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:11.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:11.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:11.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:11.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-12 zones={a2, a1} 2023-10-12T18:51:11.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:11.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:11.974 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:12.022 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:12.023 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:12.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:12.062+0000 7fd83b35f500 1 waiting for incremental sync to catch up: 2023-10-12T18:51:12.063 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000002.7.11, 00000000010.22.6, ] 2023-10-12T18:51:12.064 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-10-12T18:51:17.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:17.064+0000 7fd83b35f500 1 bucket sync caught up with source: 2023-10-12T18:51:17.065 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000003.10.6, 00000000011.26.6, ] 2023-10-12T18:51:17.065 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-10-12T18:51:17.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:17.064+0000 7fd83b35f500 0 bucket checkpoint complete 2023-10-12T18:51:17.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:17.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:17.083 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:17.134 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:17.135 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:17.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:17.177+0000 7fd8f825a500 1 bucket sync caught up with source: 2023-10-12T18:51:17.178 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000002.95.11, , , 00000000011.134.6, 00000000003.116.6, ] 2023-10-12T18:51:17.178 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.95.11, , , 00000000011.134.6, 00000000003.116.6, ] 2023-10-12T18:51:17.178 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:17.177+0000 7fd8f825a500 0 bucket checkpoint complete 2023-10-12T18:51:17.190 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-11 zones={a2, a1} 2023-10-12T18:51:17.195 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:51:17.198 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:17.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T18:51:17.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T18:51:17.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:51:17.202 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:17.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T18:51:17.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T18:51:17.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:17.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:17.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:17.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:17.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:17.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:17.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:17.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:17.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:17.465 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-11 zones={a2, a1} 2023-10-12T18:51:17.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:17.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:17.466 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:17.515 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:17.515 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:17.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:17.555+0000 7f8494a55500 1 bucket sync caught up with source: 2023-10-12T18:51:17.556 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000002.4.11, , , 00000000003.16.6, 00000000011.28.6, ] 2023-10-12T18:51:17.557 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.4.11, , , 00000000003.16.6, 00000000011.28.6, ] 2023-10-12T18:51:17.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:17.555+0000 7f8494a55500 0 bucket checkpoint complete 2023-10-12T18:51:17.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:17.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:17.570 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:17.618 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:17.618 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:17.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:17.657+0000 7ff2d42c7500 1 bucket sync caught up with source: 2023-10-12T18:51:17.658 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.76.10, , , 00000000011.261.6, 00000000003.199.6, ] 2023-10-12T18:51:17.659 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000003.76.10, , , 00000000011.261.6, 00000000003.199.6, ] 2023-10-12T18:51:17.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:17.657+0000 7ff2d42c7500 0 bucket checkpoint complete 2023-10-12T18:51:17.670 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-12 zones={a2, a1} 2023-10-12T18:51:17.675 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:51:17.678 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:17.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T18:51:17.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T18:51:17.679 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:51:17.681 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:17.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T18:51:17.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T18:51:17.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:17.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:17.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:17.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:17.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:17.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T18:51:17.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:17.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:17.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T18:51:17.900 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-12 zones={a2, a1} 2023-10-12T18:51:17.901 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-10-12T18:51:17.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:17.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:17.919 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:17.960 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:17.960 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:17.994 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:51:18.005 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:51:18.006 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:18.006 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136677.918001_72.1", 2023-10-12T18:51:18.006 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:51:17.918001Z" 2023-10-12T18:51:18.006 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:18.007 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:18.007 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:18.007 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:51:18.007 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:18.007 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:18.008 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:18.008 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:51:18.008 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:18.008 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:18.008 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:18.008 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:51:18.009 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:18.009 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:51:18.010 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:51:18.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:18.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:18.011 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:18.091 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:18.092 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:18.140 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:18.141 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:18.141 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:18.141 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:18.142 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:18.142 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:51:18.142 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:18.142 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:18.142 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:18.143 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:18.143 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:18.143 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:18.143 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:18.143 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136677.906729_67.1", 2023-10-12T18:51:18.143 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:18.144 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:51:18.144 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:18.144 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:17.906729Z", 2023-10-12T18:51:18.144 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:18.144 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:18.145 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:18.145 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:18.145 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:18.145 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:18.145 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:51:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:18.150 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:18.150 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:18.150 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:18.150 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:18.150 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:18.151 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:18.151 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:18.151 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:18.151 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:18.151 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:51:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:51:18.153 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:18.153 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:18.154 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136677.918001_72.1 target=1_1697136677.906729_67.1 2023-10-12T18:51:23.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:23.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:23.157 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:23.206 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:23.207 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:23.259 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:51:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136677.918001_72.1", 2023-10-12T18:51:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:51:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:17.918001Z", 2023-10-12T18:51:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:51:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:51:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:51:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:23.271 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:23.272 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:51:23.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:23.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:23.320 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:23.370 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:23.370 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:23.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:23.409+0000 7fc2fabc3500 1 waiting for incremental sync to catch up: 2023-10-12T18:51:23.411 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , , , ] 2023-10-12T18:51:23.412 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.7.11, , , , , ] 2023-10-12T18:51:28.412 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:28.411+0000 7fc2fabc3500 1 bucket sync caught up with source: 2023-10-12T18:51:28.413 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.9.10, , , , , ] 2023-10-12T18:51:28.413 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.7.11, , , , , ] 2023-10-12T18:51:28.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:28.411+0000 7fc2fabc3500 0 bucket checkpoint complete 2023-10-12T18:51:28.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:28.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:28.429 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:28.481 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:28.481 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:28.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:28.523+0000 7f4f77026500 1 bucket sync caught up with source: 2023-10-12T18:51:28.525 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000002.147.11, , , , , ] 2023-10-12T18:51:28.525 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.147.11, , , , , ] 2023-10-12T18:51:28.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:28.523+0000 7f4f77026500 0 bucket checkpoint complete 2023-10-12T18:51:28.537 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-13 zones={a2, a1} 2023-10-12T18:51:28.542 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:51:28.545 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:28.545 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:51:28.548 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:28.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:28.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:28.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:28.665 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-13 zones={a2, a1} 2023-10-12T18:51:29.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:29.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:29.077 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:29.127 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:29.128 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:29.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:29.170+0000 7f2073edd500 1 waiting for incremental sync to catch up: 2023-10-12T18:51:29.172 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000013.33.6, , ] 2023-10-12T18:51:29.173 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000026.69.12, , ] 2023-10-12T18:51:34.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:34.172+0000 7f2073edd500 1 bucket sync caught up with source: 2023-10-12T18:51:34.174 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000031.83.9, , ] 2023-10-12T18:51:34.174 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000026.69.12, , ] 2023-10-12T18:51:34.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:34.172+0000 7f2073edd500 0 bucket checkpoint complete 2023-10-12T18:51:34.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:34.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:34.192 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:34.245 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:34.245 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:34.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:34.288+0000 7f8459983500 1 bucket sync caught up with source: 2023-10-12T18:51:34.290 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , 00000000002.147.11, , , 00000000031.382.12, , ] 2023-10-12T18:51:34.290 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , 00000000002.147.11, , , 00000000031.382.12, , ] 2023-10-12T18:51:34.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:34.288+0000 7f8459983500 0 bucket checkpoint complete 2023-10-12T18:51:34.303 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-13 zones={a2, a1} 2023-10-12T18:51:34.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:51:34.314 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:34.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.321 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:51:34.325 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T18:51:34.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:34.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:34.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:34.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T18:51:34.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:34.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:35.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:36.385 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-13 zones={a2, a1} 2023-10-12T18:51:36.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.386 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:36.428 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:36.428 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:36.479 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:51:36.479 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.479 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000002241", 2023-10-12T18:51:36.479 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:51:33.554275Z" 2023-10-12T18:51:36.480 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.480 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.480 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000000905", 2023-10-12T18:51:36.480 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:27.577317Z" 2023-10-12T18:51:36.480 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.481 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000000643", 2023-10-12T18:51:36.481 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:27.577307Z" 2023-10-12T18:51:36.481 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.481 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000001967", 2023-10-12T18:51:36.482 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:51:33.555899Z" 2023-10-12T18:51:36.482 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.482 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:51:36.483 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T18:51:36.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.485 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:36.567 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:36.567 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:36.615 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:36.615 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:36.615 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:36.616 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:36.616 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:36.616 INFO:teuthology.orchestra.run.smithi170.stdout: "instance_id": 3025845112305917080 2023-10-12T18:51:36.616 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.617 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:36.617 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.617 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:36.617 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:36.617 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "incremental-sync", 2023-10-12T18:51:36.617 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000002241", 2023-10-12T18:51:36.618 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:36.618 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:36.618 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:36.618 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:33.554275Z" 2023-10-12T18:51:36.618 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.619 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.619 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.619 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:36.619 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:36.619 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "incremental-sync", 2023-10-12T18:51:36.620 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000000905", 2023-10-12T18:51:36.620 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:36.620 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:36.620 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:36.620 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:27.577317Z" 2023-10-12T18:51:36.621 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.621 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.621 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.621 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:36.621 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:36.621 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "incremental-sync", 2023-10-12T18:51:36.622 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000000643", 2023-10-12T18:51:36.622 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:36.622 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:36.622 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:36.622 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:27.577307Z" 2023-10-12T18:51:36.623 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.623 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.623 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.623 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:36.623 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:36.624 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "incremental-sync", 2023-10-12T18:51:36.624 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000001967", 2023-10-12T18:51:36.624 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:36.624 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:36.624 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:36.625 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:33.555899Z" 2023-10-12T18:51:36.625 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.625 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.625 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:36.625 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.625 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:36.626 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 0, 2023-10-12T18:51:36.626 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 0 2023-10-12T18:51:36.626 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.626 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:36.629 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T18:51:36.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.630 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:36.708 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:36.708 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:36.759 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:51:36.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.760 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000002241", 2023-10-12T18:51:36.760 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:51:33.532438Z" 2023-10-12T18:51:36.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.761 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000000905", 2023-10-12T18:51:36.761 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:27.901387Z" 2023-10-12T18:51:36.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.761 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000000643", 2023-10-12T18:51:36.762 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:27.978416Z" 2023-10-12T18:51:36.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.762 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000001967", 2023-10-12T18:51:36.763 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:51:33.534220Z" 2023-10-12T18:51:36.763 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.763 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:51:36.764 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T18:51:36.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.766 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:36.847 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:36.848 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:36.896 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:36.896 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:36.896 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:36.896 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:36.897 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:36.897 INFO:teuthology.orchestra.run.smithi170.stdout: "instance_id": 11900154170517088772 2023-10-12T18:51:36.897 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.897 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:36.897 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.898 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:36.898 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:36.898 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "incremental-sync", 2023-10-12T18:51:36.898 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000002241", 2023-10-12T18:51:36.898 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:36.899 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:36.899 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:36.899 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:33.532438Z" 2023-10-12T18:51:36.899 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.900 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.900 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:36.900 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:36.900 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "incremental-sync", 2023-10-12T18:51:36.900 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000000905", 2023-10-12T18:51:36.901 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:36.901 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:36.901 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:36.901 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:27.901387Z" 2023-10-12T18:51:36.901 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.902 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.902 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.902 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:36.902 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:36.902 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "incremental-sync", 2023-10-12T18:51:36.903 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000000643", 2023-10-12T18:51:36.903 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:36.903 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:36.903 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:36.903 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:27.978416Z" 2023-10-12T18:51:36.904 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:36.904 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:36.904 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:36.905 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "incremental-sync", 2023-10-12T18:51:36.905 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "00000000000000000000:00000000000000001967", 2023-10-12T18:51:36.905 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:36.905 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:36.905 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:36.906 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:33.534220Z" 2023-10-12T18:51:36.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.906 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:36.906 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:36.907 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:36.907 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 0, 2023-10-12T18:51:36.907 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 0 2023-10-12T18:51:36.907 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:36.907 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:36.908 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T18:51:36.909 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-10-12T18:51:36.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:36.924 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:36.990 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:36.990 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:37.024 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:51:37.036 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:51:37.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:37.037 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136696.921132_79.1", 2023-10-12T18:51:37.037 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:51:36.921132Z" 2023-10-12T18:51:37.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:37.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:37.037 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:37.038 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:51:37.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:37.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:37.038 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:37.039 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:51:37.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:37.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:37.039 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:37.039 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:51:37.040 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:37.040 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:51:37.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:51:37.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:37.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:37.043 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:37.124 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:37.124 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:37.172 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:37.172 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:37.173 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:37.173 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:37.173 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:37.173 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:51:37.174 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:37.174 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:37.175 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:37.175 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:37.175 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:37.176 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:37.176 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:37.176 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136696.915344_75.1", 2023-10-12T18:51:37.177 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:37.177 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:51:37.178 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:37.178 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:36.915344Z", 2023-10-12T18:51:37.178 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:37.179 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:37.179 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:37.180 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:37.180 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:37.181 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:37.181 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:37.182 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:37.182 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:37.182 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:37.183 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:37.184 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:51:37.184 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:37.184 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:37.185 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:37.185 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:37.185 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:37.186 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:37.186 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:37.187 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:37.187 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:37.187 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:37.188 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:37.188 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:37.188 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:37.189 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:37.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:37.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:37.190 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:37.190 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:37.191 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:37.191 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:37.191 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:37.192 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:37.192 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:37.192 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:37.193 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:37.193 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:37.193 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:37.194 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:37.194 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:37.195 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:37.195 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:51:37.196 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:51:37.196 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:37.196 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:37.198 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136696.921132_79.1 target=1_1697136696.915344_75.1 2023-10-12T18:51:42.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:42.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:42.201 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:42.250 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:42.250 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:42.302 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:51:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136696.921132_79.1", 2023-10-12T18:51:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:51:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:36.921132Z", 2023-10-12T18:51:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:51:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:51:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:51:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:42.315 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:42.316 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:51:42.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:42.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:42.325 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:42.377 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:42.378 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:42.418 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:42.416+0000 7fe329d48500 1 waiting for incremental sync to catch up: 2023-10-12T18:51:42.418 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , , , ] 2023-10-12T18:51:42.418 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000001.13.6, , ] 2023-10-12T18:51:47.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:47.417+0000 7fe329d48500 1 bucket sync caught up with source: 2023-10-12T18:51:47.420 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000001.13.6, , ] 2023-10-12T18:51:47.420 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000001.13.6, , ] 2023-10-12T18:51:47.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:47.417+0000 7fe329d48500 0 bucket checkpoint complete 2023-10-12T18:51:47.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:47.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:47.436 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:47.491 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:47.491 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:47.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:47.533+0000 7fd4d073c500 1 bucket sync caught up with source: 2023-10-12T18:51:47.535 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000001.160.6, , ] 2023-10-12T18:51:47.535 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000001.160.6, , ] 2023-10-12T18:51:47.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:47.533+0000 7fd4d073c500 0 bucket checkpoint complete 2023-10-12T18:51:47.548 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-14 zones={a2, a1} 2023-10-12T18:51:47.553 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:51:47.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:47.556 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:51:47.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:47.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:47.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:47.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:47.677 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-14 zones={a2, a1} 2023-10-12T18:51:47.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:47.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:47.683 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:47.724 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:47.725 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:47.759 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:51:47.771 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:51:47.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:47.771 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136707.681765_82.1", 2023-10-12T18:51:47.771 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:51:47.681765Z" 2023-10-12T18:51:47.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:47.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:47.772 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:47.772 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:51:47.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:47.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:47.773 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:47.773 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:51:47.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:47.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:47.774 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:47.774 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:51:47.774 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:47.774 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:51:47.775 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:51:47.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:47.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:47.777 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:47.857 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:47.857 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:47.905 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:47.905 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:47.905 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:47.906 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:47.906 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:47.906 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:51:47.907 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:47.907 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:47.907 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:47.908 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:47.908 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:47.908 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:47.909 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:47.909 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136707.679347_81.1", 2023-10-12T18:51:47.909 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:51:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:47.679347Z", 2023-10-12T18:51:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:47.911 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:47.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:47.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:47.912 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:47.912 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:47.912 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:47.913 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:47.913 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:47.913 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:47.914 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:47.914 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:51:47.914 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:47.914 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:47.916 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:47.916 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:47.916 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:47.918 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:47.921 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:47.921 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:47.922 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:47.922 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:47.922 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:47.923 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:47.923 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:47.923 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:47.923 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:47.924 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:47.924 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:47.924 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:47.924 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:47.925 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:47.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:47.925 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:47.926 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:51:47.926 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:51:47.926 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:47.927 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:47.928 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136707.681765_82.1 target=1_1697136707.679347_81.1 2023-10-12T18:51:52.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:52.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:52.930 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:52.978 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:52.978 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:53.030 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:53.031 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:53.031 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:53.032 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:53.032 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:53.032 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:51:53.032 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:53.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:53.033 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:53.033 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:53.033 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:53.033 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:53.034 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:53.034 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136707.681765_82.1", 2023-10-12T18:51:53.034 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:53.034 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:51:53.035 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:53.035 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:47.681765Z", 2023-10-12T18:51:53.035 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:53.035 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:53.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:53.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:53.036 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:53.036 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:53.038 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:53.039 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:53.039 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:53.040 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:53.040 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:53.040 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:51:53.040 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:53.040 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:53.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:53.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:53.041 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:53.041 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:53.041 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:53.042 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:53.042 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:53.042 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:53.042 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:53.043 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:53.043 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:53.043 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:53.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:53.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:53.045 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:53.045 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:53.046 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:53.046 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:53.046 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:53.046 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:53.047 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:53.047 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:53.047 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:53.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:53.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:53.048 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:53.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:53.048 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:53.048 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:51:53.048 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:51:53.049 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:53.049 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:53.050 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:51:53.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:53.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:53.063 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:53.114 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:53.114 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:53.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:53.153+0000 7facc11b7500 1 waiting for incremental sync to catch up: 2023-10-12T18:51:53.156 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000001.13.6, , ] 2023-10-12T18:51:53.156 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000003.16.13, , ] 2023-10-12T18:51:58.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:58.155+0000 7facc11b7500 1 bucket sync caught up with source: 2023-10-12T18:51:58.157 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000003.16.13, , ] 2023-10-12T18:51:58.157 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000003.16.13, , ] 2023-10-12T18:51:58.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:58.155+0000 7facc11b7500 0 bucket checkpoint complete 2023-10-12T18:51:58.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:58.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:51:58.174 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:51:58.228 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:58.228 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:58.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:58.270+0000 7fc6cdaad500 1 bucket sync caught up with source: 2023-10-12T18:51:58.273 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000003.173.13, , ] 2023-10-12T18:51:58.273 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000003.173.13, , ] 2023-10-12T18:51:58.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:51:58.270+0000 7fc6cdaad500 0 bucket checkpoint complete 2023-10-12T18:51:58.285 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-14 zones={a2, a1} 2023-10-12T18:51:58.290 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:51:58.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:58.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:58.340 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:51:58.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:58.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:51:58.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:58.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:58.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:58.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:58.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:58.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:51:58.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-14 zones={a2, a1} 2023-10-12T18:51:58.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:58.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:58.541 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:58.582 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:58.583 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:58.616 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:51:58.629 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:51:58.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:58.630 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136718.539578_85.1", 2023-10-12T18:51:58.630 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:51:58.539578Z" 2023-10-12T18:51:58.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:58.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:58.631 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:58.631 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:51:58.631 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:58.631 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:58.631 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:58.632 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:51:58.632 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:58.632 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:58.632 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:58.632 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:51:58.633 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:58.633 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:51:58.633 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:51:58.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:58.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:51:58.634 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:51:58.712 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:51:58.712 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:51:58.760 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:51:58.760 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:51:58.760 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:51:58.761 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:51:58.761 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:51:58.761 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:51:58.761 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:58.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:58.762 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:51:58.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:58.762 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:51:58.762 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:58.762 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:58.763 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136718.537044_84.1", 2023-10-12T18:51:58.763 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:58.763 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:51:58.763 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:58.763 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:58.537044Z", 2023-10-12T18:51:58.764 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:58.764 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:58.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:58.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:58.764 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:51:58.765 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:58.765 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:58.765 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:51:58.765 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:58.765 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:58.766 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:58.766 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:51:58.766 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:58.766 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:58.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:58.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:58.767 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:51:58.767 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:58.767 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:58.767 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:58.768 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:58.768 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:58.768 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:58.768 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:58.768 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:58.769 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:58.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:58.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:51:58.769 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:51:58.769 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:51:58.769 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:51:58.770 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:51:58.770 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:51:58.770 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:51:58.770 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:51:58.771 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:51:58.771 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:51:58.771 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:58.771 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:58.771 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:51:58.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:51:58.772 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:51:58.772 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:51:58.772 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:51:58.773 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:51:58.774 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:51:58.775 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136718.539578_85.1 target=1_1697136718.537044_84.1 2023-10-12T18:52:03.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:03.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:03.777 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:03.824 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:03.824 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:03.875 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:03.875 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:03.876 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:03.876 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:03.876 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:03.877 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:03.877 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:03.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:03.877 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:03.877 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:03.878 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:03.878 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:03.878 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:03.878 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136718.539578_85.1", 2023-10-12T18:52:03.879 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:03.879 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:03.879 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:03.879 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:51:58.539578Z", 2023-10-12T18:52:03.879 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:03.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:03.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:03.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:03.880 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:03.880 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:03.881 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:03.881 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:03.881 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:03.881 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:03.882 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:03.882 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:03.882 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:03.882 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:03.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:03.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:03.883 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:03.883 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:03.883 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:03.884 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:03.884 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:03.884 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:03.884 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:03.884 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:03.890 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:03.891 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:52:03.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:03.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:03.905 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:03.954 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:03.954 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:03.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:03.995+0000 7f4b571a5500 1 bucket sync caught up with source: 2023-10-12T18:52:03.997 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000005.20.18, , ] 2023-10-12T18:52:03.997 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000005.20.18, , ] 2023-10-12T18:52:03.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:03.995+0000 7f4b571a5500 0 bucket checkpoint complete 2023-10-12T18:52:04.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:04.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:04.010 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:04.058 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:04.059 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:04.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:04.097+0000 7f40b4be3500 1 waiting for incremental sync to catch up: 2023-10-12T18:52:04.099 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000003.173.13, , ] 2023-10-12T18:52:04.100 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000005.187.18, , ] 2023-10-12T18:52:09.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:09.099+0000 7f40b4be3500 1 bucket sync caught up with source: 2023-10-12T18:52:09.101 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000005.187.18, , ] 2023-10-12T18:52:09.101 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000005.187.18, , ] 2023-10-12T18:52:09.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:09.099+0000 7f40b4be3500 0 bucket checkpoint complete 2023-10-12T18:52:09.116 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-14 zones={a2, a1} 2023-10-12T18:52:09.122 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:52:09.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:09.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:09.126 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:52:09.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:09.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:09.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:09.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:09.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:09.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:09.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:09.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:09.285 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-14 zones={a2, a1} 2023-10-12T18:52:09.286 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-10-12T18:52:09.287 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-15 2023-10-12T18:52:09.301 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-16 2023-10-12T18:52:09.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:09.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:09.340 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:09.380 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:09.380 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:09.415 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:52:09.427 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:52:09.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:09.427 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136729.336169_97.1", 2023-10-12T18:52:09.428 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:52:09.336169Z" 2023-10-12T18:52:09.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:09.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:09.428 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:09.428 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:52:09.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:09.429 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:09.429 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:09.429 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:52:09.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:09.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:09.430 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:09.430 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:52:09.430 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:09.430 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:52:09.431 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:52:09.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:09.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:09.433 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:09.513 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:09.513 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:09.560 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:09.560 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:09.560 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:09.561 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:09.561 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:09.561 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:09.561 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:09.561 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:09.562 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:09.562 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:09.562 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:09.562 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:09.562 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:09.563 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136729.293106_86.1", 2023-10-12T18:52:09.563 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:09.563 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:09.563 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:09.564 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:09.293106Z", 2023-10-12T18:52:09.564 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:09.564 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:09.564 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:09.564 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:09.565 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:09.565 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:09.565 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:09.565 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:09.565 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:09.566 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:09.566 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:09.566 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:09.566 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:09.566 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:09.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:09.567 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:09.567 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:09.567 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:09.567 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:09.568 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:09.568 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:09.568 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:09.568 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:09.568 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:09.569 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:09.569 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:09.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:09.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:09.569 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:09.570 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:09.570 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:09.570 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:09.570 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:09.570 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:09.571 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:09.571 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:09.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:09.571 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:09.571 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:09.571 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:09.572 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:09.572 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:09.572 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:09.572 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:09.572 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:09.573 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:09.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136729.336169_97.1 target=1_1697136729.293106_86.1 2023-10-12T18:52:14.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:14.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:14.577 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:14.625 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:14.626 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:14.679 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136729.336169_97.1", 2023-10-12T18:52:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:09.336169Z", 2023-10-12T18:52:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:14.693 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:14.694 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:52:14.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:14.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:14.772 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:14.823 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:14.823 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:14.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:14.864+0000 7f855e07a500 1 bucket sync caught up with source: 2023-10-12T18:52:14.866 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000003.9.12, , ] 2023-10-12T18:52:14.866 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000003.9.12, , ] 2023-10-12T18:52:14.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:14.864+0000 7f855e07a500 0 bucket checkpoint complete 2023-10-12T18:52:14.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:14.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:14.879 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:14.928 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:14.972 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:14.970+0000 7f77e3127500 1 waiting for incremental sync to catch up: 2023-10-12T18:52:14.972 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , , , ] 2023-10-12T18:52:14.972 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000002.134.6, , ] 2023-10-12T18:52:19.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:19.971+0000 7f77e3127500 1 bucket sync caught up with source: 2023-10-12T18:52:19.974 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000002.134.6, , ] 2023-10-12T18:52:19.974 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000002.134.6, , ] 2023-10-12T18:52:19.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:19.971+0000 7f77e3127500 0 bucket checkpoint complete 2023-10-12T18:52:19.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-15 zones={a2, a1} 2023-10-12T18:52:19.995 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:52:19.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:19.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:19.999 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:52:20.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:20.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:20.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:20.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:20.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:20.119 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-15 zones={a2, a1} 2023-10-12T18:52:20.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:20.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:20.121 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:20.175 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:20.176 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:20.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:20.217+0000 7fc471256500 1 bucket sync caught up with source: 2023-10-12T18:52:20.219 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000002.15.6, , ] 2023-10-12T18:52:20.219 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000002.15.6, , ] 2023-10-12T18:52:20.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:20.217+0000 7fc471256500 0 bucket checkpoint complete 2023-10-12T18:52:20.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:20.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:20.232 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:20.279 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:20.280 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:20.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:20.319+0000 7f2ab436a500 1 bucket sync caught up with source: 2023-10-12T18:52:20.321 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000003.170.12, , ] 2023-10-12T18:52:20.321 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000003.170.12, , ] 2023-10-12T18:52:20.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:20.319+0000 7f2ab436a500 0 bucket checkpoint complete 2023-10-12T18:52:20.333 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-16 zones={a2, a1} 2023-10-12T18:52:20.339 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:52:20.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:20.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:20.346 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:52:20.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:20.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:20.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:20.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:20.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:20.422 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-16 zones={a2, a1} 2023-10-12T18:52:20.422 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-10-12T18:52:20.423 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-17 2023-10-12T18:52:20.436 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-18 2023-10-12T18:52:20.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:20.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:20.482 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:20.522 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:20.523 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:20.560 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:52:20.572 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:52:20.572 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:20.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136740.478424_116.1", 2023-10-12T18:52:20.572 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:52:20.478424Z" 2023-10-12T18:52:20.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:20.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:20.573 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:20.573 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:52:20.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:20.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:20.574 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:20.574 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:52:20.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:20.575 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:20.575 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:20.575 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:52:20.575 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:20.576 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:52:20.577 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:52:20.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:20.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:20.579 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:20.659 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:20.659 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:20.706 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:20.706 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:20.707 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:20.707 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:20.707 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:20.709 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:20.709 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:20.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:20.710 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:20.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:20.710 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:20.710 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:20.711 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:20.711 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136740.428576_101.1", 2023-10-12T18:52:20.711 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:20.711 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:20.711 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:20.712 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:20.428576Z", 2023-10-12T18:52:20.712 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:20.712 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:20.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:20.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:20.712 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:20.713 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:20.713 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:20.713 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:20.713 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:20.713 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:20.714 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:20.714 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:20.714 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:20.714 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:20.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:20.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:20.715 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:20.715 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:20.715 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:20.715 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:20.716 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:20.716 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:20.716 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:20.716 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:20.716 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:20.717 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:20.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:20.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:20.717 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:20.717 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:20.718 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:20.718 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:20.718 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:20.718 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:20.719 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:20.719 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:20.720 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:20.720 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:20.720 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:20.720 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:20.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:20.722 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:20.722 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:20.722 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:20.723 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:20.723 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:20.724 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136740.478424_116.1 target=1_1697136740.428576_101.1 2023-10-12T18:52:25.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:25.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:25.727 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:25.774 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:25.774 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:25.826 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:25.827 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:25.827 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:25.827 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:25.827 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:25.828 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:25.828 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:25.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:25.828 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:25.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:25.829 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:25.829 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:25.829 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:25.829 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136740.478424_116.1", 2023-10-12T18:52:25.829 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:25.830 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:25.830 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:25.830 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:20.478424Z", 2023-10-12T18:52:25.830 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:25.830 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:25.831 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:25.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:25.831 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:25.831 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:25.831 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:25.831 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:25.832 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:25.832 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:25.832 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:25.833 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:25.833 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:25.833 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:25.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:25.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:25.834 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:25.834 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:25.834 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:25.835 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:25.835 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:25.835 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:25.835 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:25.835 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:25.836 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:25.836 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:25.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:25.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:25.836 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:25.836 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:25.837 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:25.837 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:25.837 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:25.837 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:25.837 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:25.838 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:25.838 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:25.838 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:25.838 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:25.838 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:25.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:25.839 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:25.839 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:25.839 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:25.839 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:25.840 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:25.841 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T18:52:25.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:25.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:25.880 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:25.929 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:25.930 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:25.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:25.969+0000 7f6c65eea500 1 waiting for incremental sync to catch up: 2023-10-12T18:52:25.971 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , , , ] 2023-10-12T18:52:25.972 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000003.20.7, , ] 2023-10-12T18:52:30.972 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:30.970+0000 7f6c65eea500 1 bucket sync caught up with source: 2023-10-12T18:52:30.973 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000003.20.7, , ] 2023-10-12T18:52:30.973 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000003.20.7, , ] 2023-10-12T18:52:30.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:30.970+0000 7f6c65eea500 0 bucket checkpoint complete 2023-10-12T18:52:30.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:30.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:30.989 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:31.042 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:31.042 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:31.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:31.084+0000 7fc312fc0500 1 bucket sync caught up with source: 2023-10-12T18:52:31.086 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000001.343.13, , ] 2023-10-12T18:52:31.086 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000001.343.13, , ] 2023-10-12T18:52:31.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:31.084+0000 7fc312fc0500 0 bucket checkpoint complete 2023-10-12T18:52:31.098 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-17 zones={a2, a1} 2023-10-12T18:52:31.103 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:52:31.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:31.106 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:52:31.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:31.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:31.116 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-17 zones={a2, a1} 2023-10-12T18:52:31.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:31.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:31.118 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:31.165 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:31.166 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:31.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:31.206+0000 7f7214afb500 1 bucket sync caught up with source: 2023-10-12T18:52:31.208 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000001.11.13, , ] 2023-10-12T18:52:31.208 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000001.11.13, , ] 2023-10-12T18:52:31.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:31.206+0000 7f7214afb500 0 bucket checkpoint complete 2023-10-12T18:52:31.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:31.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T18:52:31.221 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T18:52:31.269 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:31.270 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:31.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:31.308+0000 7efc1d622500 1 bucket sync caught up with source: 2023-10-12T18:52:31.310 INFO:teuthology.orchestra.run.smithi170.stderr: local status: [, , , , , , , , 00000000003.369.7, , ] 2023-10-12T18:52:31.310 INFO:teuthology.orchestra.run.smithi170.stderr: remote markers: [, , , , , , , , 00000000003.369.7, , ] 2023-10-12T18:52:31.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T18:52:31.308+0000 7efc1d622500 0 bucket checkpoint complete 2023-10-12T18:52:31.322 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pqrzfz-18 zones={a2, a1} 2023-10-12T18:52:31.327 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T18:52:31.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:31.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T18:52:31.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T18:52:31.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T18:52:31.382 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pqrzfz-18 zones={a2, a1} 2023-10-12T18:52:31.384 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-10-12T18:52:31.384 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-19 2023-10-12T18:52:31.397 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-20 2023-10-12T18:52:31.420 INFO:rgw_multi.tests:create bucket zone=b1 name=pqrzfz-21 2023-10-12T18:52:31.455 INFO:rgw_multi.tests:create bucket zone=b2 name=pqrzfz-22 2023-10-12T18:52:31.561 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2023-10-12T18:52:31.562 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-23 2023-10-12T18:52:31.576 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-24 2023-10-12T18:52:31.599 INFO:rgw_multi.tests:create bucket zone=b1 name=pqrzfz-25 2023-10-12T18:52:31.632 INFO:rgw_multi.tests:create bucket zone=b2 name=pqrzfz-26 2023-10-12T18:52:31.742 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2023-10-12T18:52:31.743 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-27 2023-10-12T18:52:31.757 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-28 2023-10-12T18:52:31.779 INFO:rgw_multi.tests:create bucket zone=b1 name=pqrzfz-29 2023-10-12T18:52:31.808 INFO:rgw_multi.tests:create bucket zone=b2 name=pqrzfz-30 2023-10-12T18:52:34.966 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2023-10-12T18:52:34.967 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-31 2023-10-12T18:52:34.980 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-32 2023-10-12T18:52:35.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:35.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:35.005 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:35.045 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:35.046 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:35.080 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:52:35.096 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:52:35.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:35.096 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136754.994110_201.1", 2023-10-12T18:52:35.097 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:52:34.994110Z" 2023-10-12T18:52:35.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:35.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:35.097 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:35.097 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:52:35.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:35.098 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:35.098 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:35.098 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:52:35.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:35.099 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:35.099 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:35.099 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:52:35.099 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:35.099 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:52:35.100 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:52:35.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:35.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:35.102 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:35.183 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:35.183 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:35.230 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:35.231 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:35.231 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:35.231 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:35.232 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:35.232 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:35.232 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:35.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:35.233 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:35.234 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:35.234 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:35.234 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:35.235 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:35.235 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.390279_119.1", 2023-10-12T18:52:35.235 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:35.235 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:35.236 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:35.236 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.390279Z", 2023-10-12T18:52:35.236 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:35.237 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:35.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:35.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:35.238 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:35.238 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:35.238 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:35.239 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:35.239 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:35.239 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:35.240 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:35.240 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:35.240 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:35.240 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:35.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:35.241 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:35.241 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:35.241 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:35.242 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:35.242 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:35.242 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:35.242 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:35.242 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:35.243 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:35.243 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:35.243 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:35.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:35.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:35.244 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:35.244 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:35.244 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:35.244 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:35.244 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:35.245 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:35.245 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:35.245 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:35.245 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:35.245 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:35.246 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:35.246 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:35.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:35.246 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:35.246 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:35.246 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:35.247 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:35.247 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:35.248 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.390279_119.1 2023-10-12T18:52:40.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:40.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:40.250 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:40.292 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:40.293 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:40.343 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:40.343 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:40.343 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:40.343 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:52:40.346 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:40.346 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:40.346 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:40.346 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:52:40.347 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:40.347 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:40.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:40.347 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:40.347 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:40.356 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:40.356 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:52:45.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:45.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:45.359 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:45.453 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:45.453 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:45.453 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:45.454 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:45.454 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:45.454 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:45.454 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:45.454 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:45.455 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:45.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:45.455 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:45.455 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:45.455 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:45.456 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:52:45.456 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:45.456 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:45.456 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:45.456 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:52:45.457 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:45.457 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:45.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:45.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:45.457 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:45.458 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:45.458 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:45.458 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:45.458 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:45.459 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:45.459 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:45.459 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:45.459 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:45.459 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:45.460 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:45.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:45.461 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:45.462 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:45.462 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:45.462 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:45.463 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:45.463 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:45.463 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:45.464 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:45.464 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:45.465 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:45.465 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:45.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:45.466 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:45.466 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:45.466 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:45.466 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:45.467 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:45.467 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:45.467 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:45.468 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:45.468 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:45.468 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:45.469 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:45.469 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:45.469 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:45.470 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:45.470 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:45.470 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:45.470 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:45.470 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:45.472 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.436071_129.1 2023-10-12T18:52:50.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:50.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:50.475 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:50.573 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:50.574 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:51.090 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:51.091 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:51.091 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:51.091 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:51.091 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:51.091 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:51.092 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:51.092 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:51.092 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:51.092 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:51.092 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:51.093 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:51.093 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:51.093 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:52:51.093 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:51.093 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:51.093 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:51.094 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:52:51.094 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:51.094 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:51.094 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:51.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:51.095 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:51.095 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:51.095 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:51.095 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:51.095 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:51.096 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:51.096 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:51.096 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:51.096 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:51.096 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:51.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:51.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:51.097 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:51.097 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:51.097 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:51.103 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:51.104 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:52:56.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:56.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:52:56.107 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:52:56.161 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:52:56.161 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:52:56.344 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:52:56.344 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:52:56.344 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:52:56.345 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:52:56.345 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:52:56.345 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:52:56.345 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:56.346 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:56.346 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:52:56.346 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:56.346 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:52:56.346 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:56.346 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:56.347 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T18:52:56.347 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:56.347 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:52:56.347 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:56.348 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T18:52:56.348 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:56.348 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:56.348 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:56.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:56.349 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:52:56.349 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:56.349 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:56.349 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:52:56.350 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:56.350 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:56.350 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:56.350 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:52:56.350 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:56.351 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:56.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:56.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:56.351 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:52:56.351 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:56.352 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:56.352 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:56.352 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:56.352 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:56.352 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:52:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:52:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:52:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:52:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:52:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:52:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:52:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:52:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:52:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:52:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:52:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:52:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:52:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:52:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:52:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:52:56.357 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:52:56.357 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.470944_133.1 2023-10-12T18:53:01.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:01.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:01.359 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:01.401 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:01.402 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:01.448 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:01.449 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:01.449 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:01.449 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:01.449 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:01.450 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:01.450 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:01.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:01.450 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:01.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:01.451 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:01.451 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:01.451 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:01.451 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T18:53:01.451 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:01.452 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:01.452 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:01.452 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T18:53:01.453 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:01.454 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:01.454 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:01.454 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:01.454 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:01.454 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:01.455 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:01.455 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:01.455 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:01.455 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:01.455 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:01.456 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:01.456 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:01.456 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:01.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:01.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:01.457 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:01.457 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:01.458 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:01.458 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:01.458 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:01.458 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:01.458 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:01.459 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:01.459 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:01.459 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:01.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:01.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:01.460 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:01.460 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:01.460 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:01.460 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:01.460 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:01.461 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:01.461 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:01.461 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:01.461 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:01.461 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:01.462 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:01.462 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:01.462 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:01.462 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:01.462 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:01.463 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:01.467 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:01.467 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:01.468 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.438188_130.1 2023-10-12T18:53:06.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:06.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:06.473 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:06.518 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:06.519 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:06.567 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:06.567 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:06.568 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:06.568 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:06.568 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:06.568 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:06.569 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:06.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:06.569 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:06.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:06.569 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:06.570 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:06.570 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:06.570 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:53:06.570 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:06.570 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:06.571 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:06.571 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:53:06.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:06.571 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:06.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:06.572 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:06.572 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:06.572 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:06.572 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:06.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:06.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:06.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:06.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:06.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:06.574 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:06.575 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:06.575 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:06.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:06.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:06.575 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:06.576 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:06.576 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:06.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:06.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:06.576 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:06.577 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:06.577 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:06.577 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:06.577 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:06.577 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:06.578 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:06.578 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:06.578 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:06.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:06.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:06.578 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:06.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:06.579 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:06.579 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:06.579 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:06.579 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:06.580 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:06.580 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.436071_129.1 2023-10-12T18:53:11.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:11.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:11.583 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:11.626 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:11.626 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:11.675 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:11.675 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:11.675 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:11.675 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:11.676 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:11.676 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:11.676 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:11.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:11.676 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:11.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:11.677 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:11.677 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:11.677 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:11.677 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:53:11.678 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:11.678 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:11.678 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:11.678 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:53:11.678 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:11.679 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:11.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:11.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:11.679 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:11.679 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:11.680 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:11.680 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:11.680 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:11.680 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:11.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:11.681 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:11.681 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:11.681 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:11.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:11.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:11.681 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:11.682 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:11.682 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:11.682 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:11.682 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:11.683 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:11.683 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:11.683 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:11.683 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:11.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:11.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:11.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:11.684 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:11.685 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:11.685 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:11.685 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:11.685 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:11.685 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:11.685 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:11.686 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:11.686 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:11.686 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:11.686 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:11.686 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:11.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:11.687 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:11.687 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:11.687 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:11.687 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:11.688 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:11.689 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.436071_129.1 2023-10-12T18:53:16.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:16.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:16.691 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:16.734 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:16.735 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:16.782 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:16.782 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:16.782 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:16.782 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:16.783 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:16.783 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:16.783 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:16.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:16.783 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:16.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:16.784 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:16.784 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:16.784 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:16.784 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.488676_136.1", 2023-10-12T18:53:16.785 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:16.785 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:16.785 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:16.785 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.488676Z", 2023-10-12T18:53:16.785 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:16.786 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:16.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:16.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:16.786 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:16.786 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:16.787 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:16.787 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:16.787 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:16.787 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:16.787 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:16.788 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:16.788 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:16.788 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:16.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:16.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:16.789 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:16.789 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:16.789 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:16.789 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:16.789 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:16.789 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:16.790 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:16.790 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:16.790 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:16.790 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:16.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:16.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:16.792 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:16.792 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:16.792 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:16.792 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:16.792 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:16.793 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:16.793 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:16.793 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:16.793 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:16.793 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:16.794 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:16.794 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:16.794 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:16.794 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:16.794 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:16.795 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:16.795 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:16.795 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:16.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.488676_136.1 2023-10-12T18:53:21.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:21.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:21.799 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:21.846 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:21.846 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:21.899 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:21.899 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:21.899 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:21.900 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:21.900 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:21.900 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:21.900 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:21.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:21.901 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:21.901 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:21.901 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:21.902 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:21.902 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:21.902 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:53:21.902 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:21.902 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:21.903 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:21.903 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:53:21.903 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:21.903 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:21.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:21.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:21.904 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:21.904 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:21.904 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:21.904 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:21.905 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:21.905 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:21.905 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:21.905 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:21.905 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:21.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:21.906 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:21.906 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:21.906 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:21.907 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:21.907 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:21.907 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:21.907 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:21.907 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:21.907 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:21.908 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:21.908 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:21.908 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:21.908 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:21.908 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:21.909 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:21.909 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:21.909 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:21.909 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:21.910 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:21.910 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:21.910 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:21.910 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:21.911 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:21.911 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:21.911 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:21.911 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:21.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:21.912 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:21.912 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:21.912 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:21.912 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:21.912 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:21.913 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.436071_129.1 2023-10-12T18:53:26.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:26.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:26.916 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:26.958 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:26.959 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:27.009 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:27.009 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:27.009 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:27.010 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:27.010 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:27.010 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:27.010 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:27.010 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:27.011 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:27.011 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:27.011 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:27.011 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:27.011 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:27.012 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:53:27.012 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:27.012 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:27.012 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:27.012 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:53:27.013 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:27.013 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:27.013 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:27.013 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:27.013 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:27.013 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:27.014 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:27.014 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:27.014 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:27.015 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:27.015 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:27.015 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:27.015 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:27.015 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:27.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:27.016 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:27.016 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:27.016 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:27.016 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:27.017 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:27.017 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:27.017 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:27.017 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:27.017 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:27.017 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:27.018 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:27.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:27.018 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:27.018 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:27.018 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:27.019 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:27.019 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:27.019 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:27.019 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:27.019 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:27.020 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:27.020 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:27.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:27.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:27.020 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:27.020 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:27.021 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:27.021 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:27.021 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:27.021 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:27.021 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:27.022 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:53:32.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:32.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:32.025 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:32.112 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:32.113 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:32.332 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:32.332 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:32.332 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:32.332 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:53:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:53:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:32.344 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:32.344 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:32.344 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:32.344 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:32.345 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:53:37.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:37.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:37.348 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:37.391 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:37.391 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:37.488 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:37.488 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:37.489 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:37.489 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:37.489 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:37.489 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:37.489 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:37.490 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:37.490 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:37.490 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:37.490 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:37.490 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:37.491 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:37.491 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:53:37.491 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:37.491 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:37.491 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:37.492 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:53:37.492 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:37.492 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:37.492 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:37.492 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:37.492 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:37.493 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:37.493 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:37.493 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:37.493 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:37.493 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:37.494 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:37.494 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:37.494 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:37.494 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:37.494 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:37.495 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:37.495 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:37.495 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:37.495 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:37.495 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:37.495 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:37.496 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:37.496 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:37.496 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:37.496 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:37.496 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:37.497 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:37.497 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:37.497 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:37.497 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:37.497 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:37.497 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:37.498 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:37.499 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:37.499 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:37.499 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:37.499 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:37.499 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:37.500 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:37.500 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:37.500 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:37.500 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:37.500 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:37.500 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:37.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:37.501 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:37.502 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.436071_129.1 2023-10-12T18:53:42.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:42.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:42.504 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:42.585 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:42.586 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:42.687 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:42.688 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:42.688 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:42.688 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:42.688 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:42.688 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:42.689 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:42.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:42.689 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:42.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:42.689 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:42.690 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:42.690 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:42.690 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T18:53:42.690 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:42.690 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:42.691 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:42.691 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T18:53:42.691 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:42.691 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:42.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:42.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:42.692 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:42.692 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:42.692 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:42.692 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:42.692 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:42.693 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:42.693 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:42.693 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:42.693 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:42.693 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:42.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:42.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:42.694 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:42.694 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:42.694 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:42.695 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:42.695 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:42.695 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:42.695 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:42.695 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:42.696 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:42.696 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:42.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:42.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:42.696 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:42.697 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:42.697 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:42.697 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:42.697 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:42.697 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:42.697 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:42.698 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:42.698 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:42.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:42.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:42.698 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:42.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:42.699 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:42.699 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:42.699 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:42.699 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:42.700 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:42.701 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.467824_131.1 2023-10-12T18:53:47.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:47.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:47.703 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:47.746 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:47.746 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:47.794 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:47.794 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:47.794 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:47.795 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:47.795 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:47.795 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:47.795 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:47.795 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:47.796 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:47.796 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:47.796 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:47.796 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:47.796 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:47.797 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:53:47.797 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:47.797 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:47.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:47.797 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:53:47.797 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:47.798 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:47.798 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:47.798 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:47.798 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:47.798 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:47.799 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:47.799 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:47.799 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:47.799 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:47.799 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:47.800 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:47.800 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:47.800 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:47.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:47.800 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:47.801 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:47.801 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:47.801 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:47.801 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:47.801 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:47.802 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:47.802 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:47.802 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:47.803 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:47.803 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:47.803 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:47.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:47.804 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:47.804 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:47.804 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:47.804 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:47.804 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:47.804 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:47.805 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:47.805 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:47.805 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:47.805 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:47.805 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:47.806 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:47.806 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:47.806 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:47.806 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:47.806 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:47.807 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:47.807 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:47.808 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:53:52.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:52.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:52.812 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:52.854 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:52.855 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:52.902 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:52.903 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:52.903 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:52.903 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:52.903 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:52.904 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:52.904 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:52.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:52.904 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:52.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:52.905 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:52.905 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:52.905 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:52.905 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:53:52.905 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:52.906 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:52.906 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:52.906 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:53:52.906 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:52.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:52.907 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:52.907 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:52.907 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:52.907 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:52.907 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:52.908 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:52.908 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:52.908 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:52.908 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:52.908 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:52.909 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:52.909 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:52.909 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:52.909 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:52.910 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:52.910 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:52.910 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:52.910 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:52.910 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:52.911 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:52.911 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:52.911 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:52.911 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:52.911 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:52.912 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:52.912 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:52.912 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:52.912 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:52.912 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:52.913 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:52.913 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:52.913 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:52.913 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:52.914 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:52.914 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:52.914 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:52.914 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:52.915 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:52.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:52.915 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:52.915 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:52.915 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:52.916 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:52.916 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:52.918 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.436071_129.1 2023-10-12T18:53:57.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:57.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:53:57.921 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:53:57.964 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:53:57.964 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:53:58.011 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:53:58.012 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:53:58.012 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:53:58.012 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:53:58.012 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:53:58.012 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:53:58.013 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:58.013 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:58.013 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:53:58.013 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:58.013 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:53:58.014 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:58.014 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:58.014 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T18:53:58.014 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:58.014 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:53:58.015 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:58.015 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T18:53:58.015 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:58.015 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:58.015 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:58.016 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:58.016 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:53:58.016 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:58.016 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:58.016 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:53:58.017 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:58.017 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:58.017 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:58.017 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:53:58.017 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:58.017 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:58.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:58.018 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:58.018 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:53:58.018 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:58.018 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:58.019 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:58.019 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:58.019 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:58.020 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:58.020 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:58.020 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:58.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:58.020 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:58.021 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:53:58.021 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:53:58.021 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:53:58.021 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:53:58.021 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:53:58.022 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:53:58.022 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:53:58.022 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:53:58.022 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:53:58.022 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:53:58.023 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:58.023 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:58.023 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:53:58.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:53:58.023 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:53:58.024 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:53:58.024 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:53:58.024 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:53:58.024 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:53:58.025 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.392700_120.1 2023-10-12T18:54:03.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:03.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:03.028 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:03.076 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:03.077 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:03.137 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:03.137 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:03.137 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:03.137 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:03.138 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:03.138 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:03.138 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:03.138 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:03.138 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:03.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:03.139 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:03.139 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:03.139 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:03.139 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T18:54:03.140 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:03.140 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:03.140 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:03.140 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T18:54:03.140 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:03.141 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:03.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:03.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:03.141 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:03.142 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:03.142 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:03.142 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:03.142 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:03.143 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:03.143 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:03.143 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:03.143 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:03.143 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:03.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:03.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:03.144 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:03.144 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:03.144 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:03.145 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:03.145 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:03.145 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:03.145 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:03.145 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:03.146 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:03.146 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:03.146 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:03.146 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:03.146 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:03.147 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:03.147 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:03.147 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:03.147 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:03.147 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:03.147 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:03.148 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:03.148 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:03.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:03.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:03.148 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:03.149 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:03.149 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:03.149 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:03.149 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:03.149 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:03.150 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:03.151 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.404891_123.1 2023-10-12T18:54:08.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:08.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:08.153 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:08.199 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:08.199 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:08.253 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:08.254 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:08.254 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:08.254 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:08.255 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:08.255 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:08.255 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:08.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:08.255 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:08.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:08.256 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:08.256 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:08.256 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:08.256 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:54:08.257 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:08.257 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:08.257 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:08.258 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:54:08.258 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:08.258 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:08.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:08.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:08.259 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:08.259 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:08.259 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:08.259 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:08.260 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:08.260 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:08.260 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:08.260 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:08.261 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:08.261 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:08.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:08.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:08.261 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:08.262 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:08.262 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:08.262 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:08.262 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:08.262 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:08.263 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:08.263 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:08.263 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:08.263 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:08.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:08.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:08.264 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:08.264 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:08.264 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:08.264 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:08.265 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:08.265 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:08.265 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:08.265 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:08.265 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:08.266 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:08.266 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:08.266 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:08.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:08.266 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:08.267 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:08.267 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:08.267 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:08.267 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:08.268 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:54:13.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:13.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:13.271 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:13.314 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:13.315 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:13.599 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:13.599 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:13.600 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:13.600 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:13.600 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:13.600 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:13.601 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:13.601 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:13.601 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:13.601 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:13.601 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:13.601 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:13.602 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:13.602 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:54:13.602 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:13.602 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:13.603 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:13.603 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:54:13.603 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:13.603 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:13.603 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:13.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:13.604 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:13.604 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:13.604 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:13.604 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:13.604 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:13.605 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:13.605 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:13.605 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:13.605 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:13.605 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:13.606 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:13.606 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:13.606 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:13.606 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:13.607 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:13.607 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:13.607 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:13.607 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:13.607 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:13.608 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:13.608 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:13.608 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:13.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:13.608 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:13.609 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:13.609 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:13.609 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:13.609 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:13.609 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:13.609 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:13.610 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:13.610 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:13.610 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:13.611 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:13.611 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:13.611 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:13.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:13.612 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:13.612 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:13.612 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:13.612 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:13.613 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:13.614 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.436071_129.1 2023-10-12T18:54:18.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:18.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:18.617 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:18.684 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:18.685 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:18.788 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:18.789 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:18.789 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:18.789 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:18.789 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:18.789 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:18.790 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:18.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:18.790 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:18.790 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:18.790 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:18.791 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:18.791 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:18.791 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T18:54:18.791 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:18.791 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:18.792 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:18.792 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T18:54:18.792 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:18.792 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:18.792 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:18.793 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:18.793 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:18.793 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:18.793 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:18.793 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:18.794 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:18.794 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:18.794 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:18.794 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:18.794 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:18.794 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:18.795 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:18.795 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:18.795 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:18.795 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:18.795 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:18.796 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:18.796 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:18.796 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:18.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:18.797 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:18.797 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:18.798 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:18.798 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:18.798 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:18.798 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:18.798 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:18.799 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:18.799 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:18.799 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:18.799 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:18.799 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:18.800 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:18.800 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:18.800 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:18.800 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:18.800 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:18.801 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:18.801 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:18.801 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:18.801 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:18.801 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:18.801 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:18.803 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.396100_122.1 2023-10-12T18:54:23.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:23.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:23.806 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:23.862 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:23.862 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:24.092 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:54:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:54:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:24.104 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:24.105 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.436071_129.1 2023-10-12T18:54:29.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:29.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:29.108 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:29.150 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:29.151 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:29.199 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:54:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:54:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:29.211 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:29.211 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:29.212 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:29.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:29.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:29.213 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:29.214 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:29.214 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:29.214 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:29.214 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:29.214 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:29.215 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:29.215 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:29.215 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:29.216 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:29.216 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:29.216 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:29.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:29.217 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:29.217 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:29.217 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:29.217 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:29.218 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:29.219 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:54:34.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:34.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:34.222 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:34.264 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:34.265 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:34.312 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:34.312 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:34.313 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:34.313 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:34.313 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:34.313 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:34.313 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:34.315 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:34.315 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:54:34.315 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:34.315 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:34.316 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:34.316 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:54:34.316 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:34.317 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:34.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:34.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:34.317 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:34.318 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:34.318 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:34.318 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:34.318 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:34.318 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:34.319 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:34.319 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:34.319 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:34.319 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:34.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:34.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:34.320 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:34.320 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:34.322 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:34.322 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:34.322 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:34.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:34.322 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:34.323 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:34.323 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:34.323 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:34.323 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:34.323 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:34.325 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:34.325 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:34.326 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:34.326 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:34.326 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:34.326 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:34.328 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:34.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:34.328 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:34.329 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:34.329 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:34.329 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:34.329 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:34.331 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:54:39.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:39.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:39.333 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:39.376 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:39.376 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:39.425 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:39.426 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:39.426 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:39.426 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:39.426 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:39.426 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:39.427 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:39.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:39.427 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:39.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:39.427 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:39.428 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:39.428 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:39.428 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T18:54:39.428 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:39.428 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:39.428 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:39.429 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T18:54:39.429 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:39.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:39.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:39.429 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:39.430 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:39.430 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:39.430 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:39.430 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:39.430 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:39.431 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:39.431 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:39.431 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:39.431 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:39.431 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:39.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:39.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:39.432 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:39.432 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:39.432 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:39.432 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:39.433 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:39.433 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:39.433 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:39.434 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:39.434 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:39.434 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:39.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:39.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:39.435 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:39.435 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:39.435 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:39.435 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:39.435 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:39.436 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:39.436 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:39.436 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:39.436 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:39.436 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:39.437 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:39.437 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:39.437 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:39.437 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:39.437 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:39.437 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:39.438 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:39.438 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:39.439 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.408247_125.1 2023-10-12T18:54:44.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:44.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:44.441 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:44.489 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:44.490 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:44.544 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:44.544 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:44.545 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:44.545 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:44.545 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:44.545 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:44.545 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:44.546 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:44.546 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:44.546 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:44.546 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:44.546 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:44.547 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:44.547 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T18:54:44.547 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:44.547 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:44.547 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:44.548 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T18:54:44.548 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:44.548 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:44.548 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:44.548 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:44.549 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:44.549 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:44.549 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:44.549 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:44.549 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:44.549 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:44.550 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:44.550 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:44.550 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:44.550 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:44.553 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:44.553 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:44.553 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:44.554 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:44.554 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:44.554 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:44.554 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:44.554 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:44.555 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:44.555 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:44.555 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:44.555 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:44.555 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:44.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:44.556 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:44.556 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:44.556 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:44.556 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:44.557 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:44.557 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:44.557 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:44.557 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:44.557 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:44.557 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:44.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:44.558 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:44.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:44.558 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:44.558 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:44.559 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:44.559 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:44.559 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:44.560 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.408247_125.1 2023-10-12T18:54:49.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:49.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:49.563 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:49.606 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:49.606 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:49.654 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:49.654 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:49.654 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:49.654 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T18:54:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T18:54:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:49.661 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:49.661 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:49.661 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:49.661 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:49.661 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:49.661 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:49.662 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:49.662 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:49.662 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:49.662 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:49.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:49.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:49.663 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:49.663 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:49.663 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:49.663 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:49.664 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:49.664 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:49.664 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:49.664 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:49.664 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:49.665 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:49.665 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:49.665 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:49.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:49.665 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:49.665 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:49.666 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:49.666 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:49.666 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:49.667 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.392700_120.1 2023-10-12T18:54:54.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:54.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:54.669 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:54.711 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:54.711 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:54.760 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:54.760 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:54.760 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:54.760 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:54.761 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:54.761 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:54.761 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:54.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:54.761 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:54.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:54.762 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:54.762 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:54.762 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:54.762 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T18:54:54.763 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:54.763 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:54.763 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:54.763 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T18:54:54.763 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:54.763 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:54.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:54.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:54.764 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:54.765 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:54.765 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:54.765 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:54.765 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:54.765 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:54.766 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:54.766 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:54.766 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:54.766 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:54.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:54.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:54.767 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:54.768 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:54.768 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:54.768 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:54.768 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:54.768 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:54.769 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:54.769 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:54.769 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:54.770 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:54.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:54.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:54.770 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:54.770 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:54.771 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:54.771 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:54.771 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:54.771 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:54.771 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:54.771 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:54.772 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:54.772 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:54.772 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:54.772 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:54.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:54.773 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:54.773 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:54.773 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:54.773 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:54.773 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:54.774 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.408247_125.1 2023-10-12T18:54:59.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:59.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:54:59.778 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:54:59.822 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:54:59.822 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:54:59.872 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:54:59.873 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:54:59.873 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:54:59.873 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:54:59.873 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:54:59.874 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:54:59.874 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:59.874 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:59.874 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:54:59.875 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:59.875 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:54:59.875 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:59.875 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:59.875 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.583416_147.1", 2023-10-12T18:54:59.876 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:59.876 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:54:59.876 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:59.876 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.583416Z", 2023-10-12T18:54:59.876 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:59.876 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:59.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:59.877 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:59.877 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:54:59.877 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:59.877 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:59.878 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:54:59.878 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:59.878 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:59.878 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:59.878 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:54:59.879 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:59.879 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:59.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:59.879 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:59.879 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:54:59.880 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:59.880 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:59.880 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:59.880 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:59.880 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:59.881 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:59.881 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:59.881 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:59.881 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:59.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:59.881 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:54:59.882 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:54:59.882 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:54:59.882 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:54:59.882 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:54:59.882 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:54:59.883 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:54:59.883 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:54:59.883 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:54:59.883 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:54:59.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:59.884 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:59.884 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:54:59.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:54:59.884 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:54:59.884 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:54:59.885 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:54:59.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:54:59.885 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:54:59.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.583416_147.1 2023-10-12T18:55:04.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:04.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:04.893 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:04.977 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:04.977 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:05.335 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:05.336 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:05.336 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:05.336 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:05.336 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:05.336 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:05.337 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:05.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:05.337 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:05.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:05.337 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:05.338 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:05.338 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:05.338 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:55:05.338 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:05.338 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:05.339 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:05.339 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:55:05.339 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:05.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:05.340 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:05.340 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:05.340 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:05.340 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:05.341 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:05.341 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:05.341 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:05.341 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:05.341 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:05.342 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:05.342 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:05.342 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:05.342 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:05.343 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:05.343 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:05.343 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:05.343 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:05.343 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:05.344 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:05.344 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:05.344 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:05.344 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:05.344 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:05.345 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:05.345 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:05.345 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:05.345 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:05.345 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:05.345 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:05.346 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:05.346 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:05.346 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:05.346 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:05.346 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:05.347 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:05.347 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:05.347 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:05.347 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:05.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:05.348 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:05.348 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:05.348 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:05.348 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:05.348 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:05.349 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:55:10.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:10.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:10.353 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:10.396 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:10.396 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:10.481 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:10.481 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:10.481 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:10.482 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:10.482 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:10.482 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:10.482 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:10.483 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:10.483 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:10.483 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:10.483 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:10.483 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:10.484 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:10.484 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T18:55:10.484 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:10.484 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:10.484 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:10.485 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T18:55:10.485 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:10.485 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:10.485 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:10.485 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:10.486 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:10.486 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:10.487 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:10.487 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:10.487 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:10.487 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:10.487 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:10.488 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:10.488 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:10.488 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:10.488 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:10.488 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:10.489 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:10.489 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:10.489 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:10.489 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:10.489 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:10.490 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:10.490 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:10.490 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:10.490 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:10.490 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:10.491 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:10.491 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:10.491 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:10.491 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:10.491 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:10.491 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:10.492 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:10.492 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:10.492 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:10.492 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:10.492 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:10.493 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:10.493 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:10.493 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:10.493 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:10.493 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:10.494 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:10.494 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:10.494 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:10.494 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:10.495 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.410419_126.1 2023-10-12T18:55:15.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:15.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:15.498 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:15.583 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:15.583 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:15.748 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:15.748 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:15.749 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:15.749 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:15.749 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:15.749 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:15.749 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:15.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:15.750 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:15.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:15.750 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:15.750 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:15.751 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:15.751 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:55:15.751 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:15.751 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:15.751 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:15.752 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:55:15.752 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:15.752 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:15.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:15.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:15.753 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:15.753 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:15.753 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:15.754 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:15.754 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:15.755 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:15.755 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:15.755 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:15.755 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:15.755 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:15.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:15.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:15.756 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:15.756 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:15.756 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:15.757 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:15.757 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:15.757 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:15.758 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:15.758 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:15.758 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:15.758 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:15.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:15.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:15.759 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:15.759 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:15.759 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:15.759 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:15.760 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:15.760 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:15.760 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:15.760 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:15.760 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:15.761 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:15.761 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:15.761 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:15.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:15.761 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:15.762 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:15.762 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:15.762 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:15.762 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:15.765 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:55:20.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:20.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:20.769 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:20.812 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:20.812 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:20.861 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:20.862 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:20.862 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:20.862 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:20.863 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:20.863 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:20.863 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:20.864 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:20.864 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:20.864 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:20.865 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:20.865 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:20.865 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:20.866 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:55:20.866 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:20.866 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:20.867 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:20.867 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:55:20.867 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:20.867 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:20.868 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:20.868 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:20.868 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:20.869 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:20.869 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:20.869 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:20.870 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:20.870 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:20.870 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:20.870 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:20.871 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:20.871 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:20.871 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:20.871 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:20.872 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:20.872 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:20.872 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:20.872 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:20.873 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:20.873 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:20.873 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:20.874 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:20.874 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:20.874 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:20.874 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:20.874 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:20.874 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:20.875 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:20.875 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:20.875 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:20.875 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:20.875 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:20.876 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:20.876 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:20.876 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:20.876 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:20.876 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:20.877 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:20.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:20.877 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:20.878 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:20.878 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:20.878 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:20.878 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:20.880 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:55:25.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:25.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:25.883 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:25.929 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:25.929 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:26.021 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:26.021 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:26.021 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:26.022 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:26.022 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:26.022 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:26.022 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:26.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:26.023 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:26.023 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:26.023 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:26.023 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:26.024 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:26.024 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T18:55:26.024 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:26.024 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:26.024 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:26.024 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T18:55:26.025 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:26.025 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:26.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:26.025 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:26.025 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:26.026 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:26.026 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:26.026 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:26.026 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:26.026 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:26.027 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:26.027 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:26.027 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:26.027 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:26.027 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:26.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:26.028 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:26.028 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:26.028 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:26.028 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:26.028 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:26.029 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:26.029 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:26.029 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:26.029 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:26.029 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:26.033 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:26.034 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.408247_125.1 2023-10-12T18:55:31.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:31.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:31.038 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:31.080 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:31.081 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:31.129 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:31.129 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:31.129 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:31.130 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:31.130 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:31.130 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:31.130 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:31.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:31.131 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:31.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:31.131 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:31.131 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:31.132 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:31.132 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.486122_135.1", 2023-10-12T18:55:31.132 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:31.132 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:31.133 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:31.133 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.486122Z", 2023-10-12T18:55:31.133 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:31.133 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:31.133 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:31.134 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:31.134 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:31.134 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:31.134 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:31.135 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:31.135 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:31.135 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:31.135 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:31.136 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:31.136 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:31.136 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:31.136 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:31.137 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:31.137 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:31.137 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:31.137 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:31.138 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:31.138 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:31.138 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:31.139 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:31.139 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:31.139 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:31.140 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:31.140 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:31.140 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:31.141 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:31.141 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:31.141 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:31.142 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:31.142 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:31.142 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:31.142 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:31.143 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:31.143 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:31.143 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:31.144 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:31.144 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:31.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:31.145 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:31.145 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:31.145 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:31.146 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:31.146 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:31.148 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.486122_135.1 2023-10-12T18:55:36.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:36.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:36.150 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:36.196 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:36.196 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:36.248 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:36.248 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:36.248 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:36.249 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:36.249 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:36.249 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:36.249 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:36.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:36.250 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:36.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:36.250 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:36.250 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:36.250 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:36.251 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.585939_148.1", 2023-10-12T18:55:36.251 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:36.251 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:36.251 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:36.251 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.585939Z", 2023-10-12T18:55:36.252 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:36.252 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:36.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:36.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:36.252 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:36.253 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:36.253 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:36.253 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:36.253 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:36.253 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:36.253 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:36.254 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:36.254 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:36.254 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:36.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:36.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:36.255 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:36.255 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:36.255 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:36.255 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:36.255 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:36.256 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:36.256 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:36.256 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:36.256 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:36.256 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:36.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:36.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:36.257 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:36.257 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:36.257 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:36.258 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:36.258 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:36.258 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:36.258 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:36.258 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:36.258 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:36.259 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:36.259 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:36.259 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:36.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:36.259 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:36.260 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:36.260 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:36.260 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:36.260 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:36.261 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.585939_148.1 2023-10-12T18:55:41.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:41.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:41.264 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:41.325 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:41.325 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:41.731 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:41.731 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:41.731 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T18:55:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T18:55:41.735 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:41.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:41.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:41.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:41.736 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:41.736 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:41.737 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:41.737 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:41.737 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:41.738 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:41.738 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:41.739 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:41.739 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:41.741 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:41.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:41.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:41.742 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:41.743 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:41.743 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:41.743 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:41.743 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:41.746 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:41.746 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:41.746 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:41.746 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:41.746 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:41.748 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:41.748 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:41.748 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:41.748 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:41.748 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:41.749 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.392700_120.1 2023-10-12T18:55:46.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:46.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:46.752 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:46.793 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:46.794 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:46.843 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:55:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:55:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:46.858 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:46.860 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:55:51.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:51.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:51.862 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:51.925 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:51.925 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:52.053 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:52.054 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:52.054 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:52.054 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:52.054 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:52.054 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:52.055 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:52.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:52.055 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:52.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:52.055 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T18:55:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T18:55:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:52.058 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:52.058 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:52.058 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:52.058 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:52.059 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:52.059 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:52.059 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:52.059 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:52.060 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:52.060 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:52.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:52.060 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:52.060 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:52.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:52.063 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:52.063 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:52.063 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:52.063 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:52.064 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:52.064 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:52.064 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:52.064 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:52.064 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:52.064 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:52.065 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:52.065 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:52.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:52.065 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:52.065 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:52.066 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:52.066 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:52.066 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:52.067 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.404891_123.1 2023-10-12T18:55:57.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:57.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:55:57.069 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:55:57.134 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:55:57.135 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:55:57.182 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:55:57.182 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:55:57.182 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:55:57.183 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:55:57.183 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:55:57.183 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:55:57.183 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:57.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:57.184 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:55:57.184 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:57.184 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:55:57.184 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:57.185 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:57.185 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:55:57.185 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:57.185 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:55:57.185 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:57.186 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:55:57.186 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:57.186 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:57.186 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:57.186 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:57.187 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:55:57.187 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:57.187 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:57.187 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:55:57.187 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:57.187 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:57.188 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:57.188 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:55:57.188 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:57.188 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:57.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:57.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:57.189 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:55:57.189 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:57.189 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:57.189 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:57.190 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:57.190 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:57.190 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:57.190 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:57.190 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:57.190 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:57.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:57.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:55:57.191 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:55:57.191 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:55:57.191 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:55:57.192 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:55:57.192 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:55:57.192 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:55:57.192 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:55:57.192 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:55:57.193 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:55:57.193 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:57.193 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:57.193 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:55:57.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:55:57.193 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:55:57.194 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:55:57.194 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:55:57.194 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:55:57.194 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:55:57.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:56:02.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:02.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:02.198 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:02.246 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:02.247 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:02.297 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:02.298 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:02.298 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:02.299 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:02.299 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:02.299 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:02.299 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:02.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:02.300 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:02.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:02.301 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:02.301 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:02.301 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:02.301 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:56:02.302 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:02.302 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:02.302 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:02.303 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:56:02.303 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:02.303 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:02.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:02.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:02.304 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:02.304 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:02.305 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:02.305 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:02.305 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:02.305 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:02.305 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:02.306 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:02.306 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:02.306 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:02.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:02.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:02.307 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:02.307 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:02.307 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:02.309 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:02.309 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:02.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:02.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:02.310 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:02.310 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:02.310 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:02.310 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:02.312 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:02.312 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:02.312 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:02.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:02.313 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:02.313 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:02.313 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:02.313 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:02.313 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:02.315 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:56:07.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:07.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:07.318 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:07.361 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:07.361 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:07.409 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:07.409 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:07.409 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:07.410 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:07.410 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:07.410 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:07.410 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:07.410 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:07.411 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:07.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:07.411 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:07.411 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:07.412 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:07.412 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:56:07.412 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:07.412 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:07.412 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:07.413 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:56:07.413 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:07.413 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:07.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:07.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:07.414 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:07.414 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:07.414 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:07.414 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:07.414 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:07.415 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:07.415 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:07.415 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:07.415 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:07.415 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:07.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:07.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:07.416 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:07.416 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:07.416 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:07.417 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:07.417 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:07.417 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:07.417 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:07.417 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:07.417 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:07.418 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:07.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:07.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:07.418 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:07.418 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:07.419 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:07.419 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:07.419 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:07.419 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:07.419 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:07.420 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:07.420 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:07.420 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:07.420 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:07.420 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:07.421 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:07.421 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:07.421 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:07.421 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:07.421 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:07.422 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:07.423 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:56:12.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:12.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:12.425 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:12.468 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:12.469 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:12.519 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:12.520 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:12.520 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:12.520 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:12.520 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:12.521 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:12.521 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:12.521 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:12.521 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:12.521 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:12.522 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:12.522 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:12.522 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:12.522 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:56:12.522 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:56:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:12.523 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:12.524 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:12.525 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:12.526 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:12.527 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:12.527 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:12.527 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:12.527 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:12.527 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:12.528 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:12.529 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:12.529 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:12.529 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:12.529 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:12.530 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:12.530 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:12.531 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:12.531 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:12.531 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:12.531 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:12.531 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:12.531 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:12.532 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:12.532 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:12.532 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:12.532 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:12.532 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:12.533 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:12.533 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:12.534 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:56:17.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:17.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:17.537 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:17.580 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:17.580 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:17.628 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:17.629 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:17.629 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:17.630 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:17.630 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:17.630 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:17.631 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:17.631 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:17.631 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:17.632 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:17.632 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:17.633 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:17.633 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:17.633 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T18:56:17.634 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:17.634 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:17.635 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:17.635 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T18:56:17.635 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:17.635 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:17.636 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:17.636 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:17.636 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:17.636 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:17.637 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:17.637 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:17.637 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:17.637 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:17.637 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:17.638 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:17.638 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:17.638 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:17.638 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:17.639 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:17.639 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:17.639 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:17.639 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:17.639 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:17.639 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:17.640 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:17.640 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:17.640 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:17.640 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:17.640 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:17.641 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:17.641 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:17.641 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:17.641 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:17.641 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:17.642 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:17.642 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:17.642 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:17.642 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:17.643 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:17.643 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:17.644 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:17.644 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:17.644 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:17.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:17.645 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:17.645 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:17.646 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:17.646 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:17.647 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:17.649 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.470944_133.1 2023-10-12T18:56:22.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:22.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:22.652 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:22.694 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:22.694 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:22.742 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:22.743 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:22.743 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:22.743 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:22.743 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:22.744 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:22.744 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:22.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:22.744 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:22.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:22.745 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:22.745 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:22.745 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:22.745 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T18:56:22.745 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:22.746 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:22.746 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:22.746 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T18:56:22.746 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:22.746 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:22.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:22.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:22.747 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:22.747 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:22.747 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:22.748 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:22.748 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:22.748 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:22.748 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:22.748 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:22.749 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:22.749 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:22.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:22.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:22.749 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:22.750 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:22.750 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:22.750 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:22.750 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:22.750 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:22.751 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:22.751 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:22.751 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:22.751 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:22.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:22.755 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:22.755 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:22.755 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:22.756 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.392700_120.1 2023-10-12T18:56:27.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:27.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:27.759 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:27.804 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:27.805 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:27.856 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:27.856 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:27.857 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:27.857 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:27.857 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:27.857 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:27.857 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:27.858 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:27.858 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:27.858 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:27.858 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:27.858 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:27.859 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:27.859 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T18:56:27.859 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:27.859 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:27.859 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:27.860 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T18:56:27.860 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:27.860 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:27.860 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:27.860 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:27.861 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:27.861 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:27.861 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:27.861 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:27.861 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:27.862 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:27.862 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:27.862 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:27.862 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:27.862 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:27.863 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:27.863 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:27.863 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:27.863 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:27.863 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:27.864 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:27.864 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:27.864 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:27.864 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:27.864 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:27.865 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:27.865 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:27.865 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:27.865 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:27.865 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:27.865 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:27.866 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:27.866 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:27.866 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:27.866 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:27.867 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:27.867 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:27.867 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:27.867 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:27.867 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:27.868 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:27.868 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:27.868 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:27.868 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:27.868 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:27.868 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:27.869 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:27.870 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.408247_125.1 2023-10-12T18:56:32.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:32.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:32.872 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:32.914 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:32.915 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:33.007 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:33.008 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:33.008 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:33.008 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:33.008 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:33.009 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:33.009 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:33.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:33.009 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:33.009 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:33.010 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:33.010 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:33.010 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:33.010 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:56:33.010 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:33.011 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:33.011 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:33.011 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:56:33.011 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:33.012 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:33.012 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:33.012 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:33.012 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:33.012 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:33.013 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:33.013 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:33.013 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:33.013 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:33.013 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:33.014 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:33.014 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:33.014 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:33.014 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:33.014 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:33.015 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:33.015 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:33.015 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:33.015 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:33.015 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:33.016 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:33.016 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:33.016 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:33.016 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:33.016 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:33.017 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:33.017 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:33.017 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:33.017 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:33.017 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:33.018 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:33.018 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:33.018 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:33.018 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:33.018 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:33.019 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:33.019 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:33.019 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:33.019 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:33.019 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:33.020 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:33.020 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:33.020 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:33.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:33.020 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:33.022 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:56:38.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:38.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:38.024 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:38.144 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:38.144 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:38.385 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:38.385 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:38.386 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:38.386 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:38.386 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:38.386 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:38.387 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:38.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:38.387 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:38.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:38.388 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:38.388 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:38.388 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:38.388 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T18:56:38.389 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:38.389 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:38.389 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:38.389 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T18:56:38.390 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:38.390 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:38.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:38.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:38.390 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:38.391 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:38.391 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:38.391 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:38.391 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:38.391 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:38.392 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:38.392 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:38.392 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:38.393 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:38.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:38.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:38.394 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:38.394 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:38.394 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:38.395 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:38.395 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:38.395 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:38.396 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:38.396 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:38.396 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:38.397 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:38.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:38.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:38.398 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:38.398 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:38.398 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:38.399 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:38.399 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:38.399 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:38.399 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:38.399 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:38.400 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:38.400 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:38.400 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:38.400 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:38.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:38.401 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:38.401 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:38.401 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:38.401 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:38.401 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:38.402 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.432502_127.1 2023-10-12T18:56:43.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:43.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:43.405 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:43.447 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:43.447 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:43.497 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:43.497 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:43.498 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:43.498 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:43.498 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:43.498 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:43.499 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:43.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:43.499 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:43.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:43.499 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:43.500 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:43.500 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:43.500 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T18:56:43.500 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:43.501 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:43.501 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:43.501 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T18:56:43.501 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:43.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:43.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:43.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:43.502 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:43.502 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:43.503 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:43.503 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:43.503 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:43.503 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:43.503 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:43.504 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:43.504 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:43.504 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:43.504 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:43.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:43.505 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:43.505 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:43.505 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:43.505 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:43.506 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:43.506 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:43.506 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:43.506 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:43.507 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:43.507 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:43.507 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:43.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:43.507 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:43.508 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:43.508 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:43.508 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:43.508 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:43.508 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:43.509 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:43.509 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:43.509 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:43.509 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:43.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:43.510 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:43.510 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:43.510 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:43.510 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:43.511 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:43.511 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:43.511 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:43.512 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.392700_120.1 2023-10-12T18:56:48.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:48.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:48.515 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:48.581 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:48.581 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:48.776 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:48.777 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:48.777 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:48.777 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:48.777 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:48.778 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:48.778 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:48.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:48.778 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:48.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:48.779 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:48.779 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:48.779 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:48.779 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:56:48.779 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:48.780 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:48.780 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:48.780 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:56:48.780 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:48.781 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:48.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:48.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:48.781 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:48.781 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:48.782 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:48.782 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:48.782 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:48.782 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:48.783 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:48.783 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:48.783 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:48.783 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:48.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:48.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:48.784 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:48.784 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:48.784 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:48.785 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:48.785 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:48.785 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:48.786 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:48.786 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:48.786 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:48.786 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:48.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:48.787 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:48.787 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:48.787 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:48.787 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:48.788 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:48.788 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:48.788 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:48.788 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:48.788 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:48.789 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:48.789 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:48.789 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:48.789 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:48.789 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:48.790 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:48.790 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:48.790 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:48.790 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:48.790 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:48.791 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:56:53.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:53.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:53.794 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:53.844 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:53.845 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:53.893 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:53.894 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:53.894 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:53.894 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:53.894 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:53.895 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:53.895 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:53.895 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:53.895 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:53.896 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:53.896 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:53.896 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:53.896 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:53.896 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.767744_174.1", 2023-10-12T18:56:53.897 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:53.897 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:53.897 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:53.897 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.767744Z", 2023-10-12T18:56:53.897 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:53.897 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:53.898 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:53.898 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:53.898 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:53.898 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:53.898 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:53.899 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:53.899 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:53.899 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:53.900 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:53.900 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:53.900 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:53.900 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:53.900 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:53.901 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:53.901 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:53.901 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:53.901 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:53.902 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:53.902 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:53.902 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:53.902 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:53.902 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:53.903 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:53.903 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:53.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:53.903 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:53.903 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:53.904 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:53.904 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:53.904 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:53.904 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:53.904 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:53.905 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:53.905 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:53.905 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:53.905 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:53.905 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:53.906 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:53.906 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:53.906 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:53.906 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:53.906 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:53.907 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:53.907 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:53.908 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.767744_174.1 2023-10-12T18:56:58.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:58.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:56:58.910 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:56:58.952 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:56:58.952 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:56:59.000 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:56:59.000 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:56:59.001 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:56:59.001 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:56:59.001 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:56:59.001 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:56:59.002 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:59.002 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:59.002 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:56:59.002 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:59.002 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:56:59.003 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:59.003 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:59.003 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:56:59.003 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:59.003 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:56:59.004 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:59.004 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:56:59.004 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:59.004 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:59.004 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:59.004 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:59.005 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:56:59.005 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:59.005 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:59.005 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:56:59.006 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:59.006 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:59.006 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:59.006 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:56:59.006 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:59.007 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:59.007 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:59.007 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:59.007 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:56:59.007 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:59.008 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:59.008 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:59.008 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:59.008 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:59.008 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:59.009 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:59.009 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:59.009 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:59.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:59.009 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:56:59.010 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:56:59.010 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:56:59.010 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:56:59.010 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:56:59.010 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:56:59.011 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:56:59.011 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:56:59.011 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:56:59.011 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:56:59.011 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:59.012 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:59.012 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:56:59.012 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:56:59.012 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:56:59.012 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:56:59.013 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:56:59.013 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:56:59.013 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:56:59.014 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:57:04.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:04.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:04.017 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:04.059 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:04.060 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:04.107 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:04.107 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:04.108 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:04.108 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:04.108 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:04.109 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:04.109 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:04.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:04.109 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:04.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:04.110 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:04.110 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:04.110 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:04.110 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T18:57:04.110 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:04.111 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:04.111 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:04.111 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T18:57:04.111 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:04.111 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:04.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:04.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:04.112 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:04.112 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:04.112 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:04.113 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:04.113 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:04.113 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:04.113 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:04.113 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:04.114 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:04.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:04.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:04.114 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:04.115 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:04.115 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:04.115 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:04.115 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:04.115 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:04.116 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:04.116 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:04.116 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:04.116 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:04.116 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:04.116 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:04.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:04.117 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:04.117 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:04.117 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:04.117 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:04.118 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:04.118 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:04.118 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:04.118 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:04.119 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:04.119 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:04.119 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:04.119 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:04.119 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:04.120 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:04.120 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:04.120 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:04.120 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:04.121 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:04.122 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.404891_123.1 2023-10-12T18:57:09.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:09.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:09.124 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:09.166 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:09.167 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:09.215 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:09.215 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:09.215 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:09.216 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:09.216 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:09.216 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:09.216 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:09.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:09.217 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:09.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:09.217 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:09.217 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:09.217 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:09.217 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:57:09.218 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:09.218 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:09.218 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:09.218 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:57:09.219 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:09.219 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:09.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:09.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:09.219 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:09.219 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:09.220 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:09.220 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:09.220 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:09.220 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:09.220 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:09.221 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:09.221 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:09.221 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:09.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:09.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:09.222 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:09.222 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:09.222 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:09.222 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:09.222 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:09.223 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:09.223 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:09.223 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:09.223 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:09.223 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:09.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:09.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:09.224 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:09.224 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:09.224 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:09.224 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:09.225 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:09.225 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:09.225 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:09.225 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:09.225 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:09.226 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:09.226 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:09.226 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:09.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:09.226 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:09.227 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:09.227 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:09.227 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:09.227 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:09.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:57:14.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:14.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:14.230 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:14.274 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:14.275 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:14.328 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:14.328 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:14.328 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:14.329 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:14.329 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:14.329 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:14.329 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:14.329 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:14.329 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:14.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:14.330 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:14.330 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:14.330 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:14.330 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T18:57:14.331 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:14.331 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:14.331 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:14.331 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T18:57:14.331 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:14.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:14.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:14.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:14.332 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:14.332 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:14.333 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:14.333 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:14.333 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:14.333 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:14.334 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:14.334 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:14.334 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:14.334 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:14.334 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:14.335 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:14.335 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:14.335 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:14.335 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:14.335 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:14.336 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:14.336 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:14.336 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:14.336 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:14.336 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:14.336 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:14.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:14.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:14.337 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:14.337 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:14.337 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:14.338 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:14.338 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:14.338 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:14.338 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:14.338 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:14.339 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:14.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:14.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:14.339 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:14.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:14.339 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:14.340 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:14.340 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:14.340 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:14.340 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:14.341 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.435183_128.1 2023-10-12T18:57:19.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:19.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:19.344 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:19.386 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:19.387 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:19.436 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:19.437 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:19.437 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:19.437 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:19.437 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:19.437 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:19.438 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:19.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:19.438 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:19.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:19.438 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:19.439 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:19.439 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:19.439 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:57:19.439 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:19.439 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:19.440 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:19.440 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:57:19.440 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:19.440 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:19.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:19.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:19.441 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:19.441 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:19.441 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:19.441 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:19.441 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:19.442 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:19.442 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:19.442 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:19.442 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:19.442 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:19.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:19.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:19.443 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:19.443 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:19.443 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:19.444 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:19.444 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:19.444 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:19.444 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:19.444 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:19.445 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:19.445 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:19.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:19.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:19.446 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:19.446 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:19.446 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:19.446 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:19.446 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:19.447 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:19.447 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:19.447 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:19.447 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:19.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:19.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:19.448 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:19.448 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:19.448 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:19.448 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:19.448 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:19.449 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:19.449 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:19.450 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:57:24.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:24.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:24.454 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:24.500 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:24.502 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:24.889 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T18:57:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T18:57:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:24.904 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:24.905 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.435183_128.1 2023-10-12T18:57:29.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:29.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:29.907 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:29.997 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:29.997 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:30.106 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:30.106 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:30.106 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:30.107 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:30.107 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:30.107 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:30.107 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:30.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:30.107 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:30.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:30.108 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:30.108 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:30.108 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:30.108 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T18:57:30.109 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:30.109 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:30.109 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:30.109 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T18:57:30.109 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:30.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:30.110 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:30.110 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:30.110 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:30.110 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:30.111 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:30.111 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:30.111 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:30.111 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:30.111 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:30.111 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:30.112 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:30.112 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:30.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:30.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:30.112 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:30.113 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:30.113 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:30.113 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:30.113 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:30.113 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:30.114 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:30.114 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:30.114 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:30.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:30.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:30.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:30.115 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:30.115 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:30.115 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:30.115 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:30.115 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:30.116 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:30.116 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:30.116 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:30.116 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:30.116 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:30.117 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:30.117 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:30.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:30.117 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:30.117 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:30.117 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:30.118 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:30.118 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:30.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.407338_124.1 2023-10-12T18:57:35.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:35.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:35.123 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:35.170 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:35.170 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:35.542 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:35.542 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:35.543 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:35.543 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:35.543 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:35.543 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:35.543 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:35.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:35.544 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:35.544 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:35.544 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:35.544 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:35.545 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:35.545 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:57:35.545 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:35.545 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:35.545 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:35.546 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:57:35.546 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:35.546 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:35.546 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:35.546 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:35.547 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:35.547 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:35.547 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:35.547 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:35.547 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:35.548 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:35.548 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:35.548 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:35.548 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:35.548 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:35.549 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:35.549 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:35.549 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:35.549 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:35.549 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:35.549 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:35.550 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:35.550 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:35.550 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:35.550 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:35.551 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:35.551 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:35.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:35.551 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:35.551 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:35.551 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:35.552 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:35.552 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:35.552 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:35.552 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:35.552 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:35.553 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:35.553 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:35.553 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:35.553 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:35.553 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:35.554 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:35.554 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:35.554 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:35.554 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:35.554 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:35.555 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:35.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.393673_121.1 2023-10-12T18:57:40.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:40.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:40.558 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:40.601 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:40.601 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:40.650 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:40.651 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:40.651 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:40.651 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:40.651 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:40.652 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:40.652 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:40.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:40.652 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:40.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:40.653 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:40.653 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:40.653 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:40.653 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:57:40.653 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:40.654 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:40.654 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:40.654 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:57:40.654 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:40.654 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:40.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:40.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:40.655 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:40.655 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:40.655 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:40.656 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:40.656 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:40.656 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:40.656 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:40.656 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:40.657 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:40.657 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:40.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:40.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:40.657 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:40.657 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:40.658 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:40.658 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:40.658 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:40.658 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:40.658 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:40.659 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:40.659 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:40.659 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:40.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:40.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:40.660 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:40.660 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:40.660 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:40.660 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:40.660 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:40.661 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:40.661 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:40.661 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:40.661 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:40.661 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:40.661 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:40.662 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:40.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:40.662 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:40.662 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:40.662 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:40.663 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:40.663 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:40.664 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697136754.994110_201.1 target=1_1697136751.436071_129.1 2023-10-12T18:57:45.666 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2023-10-12T18:57:45.666 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-33 2023-10-12T18:57:45.683 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-34 2023-10-12T18:57:45.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:45.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:45.730 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:45.773 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:45.773 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:45.792 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T18:57:45.790+0000 7f663fe6b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T18:57:45.812 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T18:57:45.826 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T18:57:45.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:45.826 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697137065.700268_210.1", 2023-10-12T18:57:45.827 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:57:45.700268Z" 2023-10-12T18:57:45.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:45.827 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:45.827 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:45.828 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T18:57:45.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:45.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:45.828 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:45.828 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:57:45.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:45.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:45.829 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:45.829 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T18:57:45.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:45.830 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T18:57:45.831 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T18:57:45.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:45.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:45.832 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:45.915 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:45.915 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:45.965 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:45.966 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:45.966 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:45.966 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:45.966 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:45.966 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:45.967 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:45.967 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:45.967 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:45.967 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:45.968 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:45.968 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:45.968 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:45.968 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:57:45.968 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:45.969 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:45.969 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:45.969 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:57:45.969 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:45.969 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:45.969 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:45.970 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:45.970 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:45.970 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:45.971 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:45.971 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:45.971 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:45.971 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:45.971 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:45.972 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:45.972 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:45.972 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:45.972 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:45.973 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:45.973 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:45.973 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:45.973 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:45.974 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:45.974 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:45.974 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:45.974 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:45.974 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:45.975 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:45.975 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:45.975 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:45.975 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:45.975 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:45.976 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:45.976 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:45.976 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:45.976 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:45.976 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:45.976 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:45.977 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:45.977 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:45.977 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:45.977 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:45.977 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:45.978 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:45.978 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:45.978 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:45.978 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:45.978 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:45.979 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:45.980 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:57:50.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:50.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:50.983 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:51.026 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:51.027 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:51.097 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:51.097 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:51.097 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:51.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:51.099 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T18:57:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:51.100 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T18:57:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:51.101 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:51.102 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:51.103 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:51.103 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:51.103 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:51.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:51.103 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:51.104 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:51.104 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:51.104 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:51.104 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:51.104 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:51.105 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:51.105 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:51.105 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:51.105 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:51.105 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:51.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:51.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:51.106 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:51.106 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:51.107 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:51.108 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:51.108 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:51.108 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:51.108 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:51.108 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:51.109 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:51.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:51.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:51.109 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:51.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:51.110 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:51.110 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:51.110 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:51.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:51.110 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:51.111 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.470944_133.1 2023-10-12T18:57:56.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:56.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:57:56.114 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:57:56.159 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:57:56.159 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:57:56.212 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:57:56.212 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:57:56.212 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:57:56.212 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:57:56.213 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:57:56.213 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:57:56.213 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:56.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:56.213 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:57:56.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:56.214 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:57:56.214 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:56.214 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:56.214 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:57:56.214 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:56.215 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:57:56.215 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:56.215 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:57:56.215 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:56.215 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:56.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:56.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:56.216 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:57:56.216 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:56.216 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:56.217 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:57:56.217 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:56.217 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:56.217 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:56.217 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:57:56.217 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:56.218 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:56.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:56.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:56.219 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:57:56.219 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:56.220 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:56.220 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:56.220 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:56.220 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:56.220 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:56.221 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:56.221 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:56.221 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:56.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:56.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:57:56.222 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:57:56.222 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:57:56.222 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:57:56.222 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:57:56.222 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:57:56.222 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:57:56.223 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:57:56.223 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:57:56.223 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:57:56.223 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:56.223 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:56.224 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:57:56.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:57:56.224 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:57:56.224 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:57:56.224 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:57:56.225 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:57:56.225 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:57:56.226 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:58:01.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:01.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:01.228 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:01.271 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:01.271 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:01.320 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:01.321 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:01.321 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:01.321 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:01.321 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:01.322 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:01.322 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:01.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:01.322 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:01.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:01.323 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:01.323 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:01.323 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:01.323 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:58:01.323 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:01.324 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:01.324 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:01.324 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:58:01.324 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:01.324 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:01.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:01.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:01.325 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:01.325 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:01.325 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:01.326 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:01.326 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:01.326 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:01.326 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:01.326 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:01.327 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:01.327 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:01.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:01.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:01.327 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:01.328 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:01.328 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:01.328 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:01.328 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:01.328 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:01.329 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:01.329 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:01.329 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:01.329 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:01.329 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:01.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:01.330 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:01.330 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:01.330 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:01.330 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:01.330 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:01.331 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:01.331 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:01.331 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:01.331 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:01.331 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:01.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:01.332 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:01.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:01.332 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:01.332 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:01.333 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:01.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:01.333 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:01.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:58:06.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:06.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:06.337 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:06.384 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:06.384 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:06.438 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:06.438 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:06.438 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:06.438 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:06.439 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:06.439 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:06.439 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:06.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:06.439 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:06.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:06.440 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:06.440 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:06.440 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:06.440 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:58:06.441 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:06.441 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:06.441 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:06.441 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:58:06.441 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:06.442 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:06.442 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:06.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:06.442 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:06.442 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:06.443 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:06.443 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:06.443 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:06.443 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:06.443 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:06.443 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:06.444 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:06.444 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:06.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:06.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:06.444 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:06.445 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:06.445 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:06.445 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:06.445 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:06.446 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:06.446 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:06.446 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:06.446 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:06.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:06.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:06.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:06.447 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:06.447 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:06.448 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:06.448 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:06.448 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:06.448 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:06.448 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:06.448 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:06.449 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:06.449 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:06.449 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:06.449 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:06.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:06.450 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:06.450 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:06.450 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:06.450 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:06.450 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:06.452 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:58:11.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:11.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:11.455 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:11.497 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:11.497 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:11.583 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:11.584 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:11.584 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:11.584 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:11.584 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:11.585 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:11.585 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:11.585 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:11.585 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:11.585 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:11.585 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:11.586 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:11.586 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:11.586 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:58:11.586 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:11.586 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:11.587 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:11.587 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:58:11.587 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:11.587 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:11.587 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:11.588 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:11.588 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:11.588 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:11.588 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:11.588 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:11.589 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:11.589 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:11.589 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:11.589 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:11.589 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:11.590 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:11.590 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:11.590 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:11.590 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:11.590 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:11.591 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:11.591 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:11.591 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:11.591 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:11.591 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:11.591 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:11.592 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:11.592 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:11.592 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:11.592 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:11.592 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:11.593 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:11.593 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:11.593 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:11.593 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:11.593 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:11.594 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:11.594 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:11.594 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:11.594 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:11.594 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:11.595 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:11.595 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:11.595 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:11.595 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:11.595 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:11.595 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:11.596 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:11.597 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:58:16.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:16.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:16.599 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:16.643 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:16.643 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:16.695 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:16.695 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:16.696 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:16.696 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:16.697 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:16.697 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:16.697 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:16.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:16.698 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:16.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:16.699 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:16.699 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:16.699 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:16.699 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:58:16.700 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:16.700 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:16.700 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:16.700 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:58:16.700 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:16.701 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:16.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:16.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:16.702 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:16.702 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:16.702 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:16.702 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:16.703 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:16.703 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:16.703 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:16.704 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:16.704 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:16.704 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:16.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:16.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:16.705 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:16.706 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:16.706 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:16.706 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:16.707 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:16.707 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:16.708 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:16.708 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:16.708 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:16.709 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:16.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:16.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:16.709 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:16.710 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:16.710 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:16.710 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:16.711 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:16.711 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:16.711 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:16.711 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:16.711 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:16.712 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:16.712 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:16.712 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:16.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:16.713 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:16.713 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:16.713 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:16.713 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:16.714 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:16.715 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:58:21.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:21.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:21.718 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:21.760 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:21.760 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:21.813 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:21.813 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:21.814 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:21.814 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:21.814 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:21.814 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:21.814 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:21.815 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:21.815 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:21.815 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:21.816 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:21.816 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:21.816 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:21.817 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T18:58:21.817 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:21.817 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:21.818 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:21.818 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T18:58:21.818 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:21.819 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:21.819 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:21.819 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:21.820 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:21.820 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:21.820 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:21.821 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:21.822 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:21.822 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:21.822 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:21.823 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:21.823 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:21.823 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:21.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:21.824 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:21.825 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:21.825 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:21.825 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:21.826 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:21.826 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:21.826 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:21.827 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:21.827 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:21.828 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:21.828 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:21.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:21.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:21.829 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:21.829 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:21.830 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:21.830 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:21.830 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:21.831 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:21.832 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:21.832 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:21.832 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:21.833 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:21.833 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:21.834 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:21.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:21.834 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:21.835 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:21.835 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:21.835 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:21.836 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:21.838 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.408247_125.1 2023-10-12T18:58:26.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:26.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:26.841 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:26.883 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:26.884 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:26.933 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:26.934 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:26.934 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:26.934 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:26.934 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:26.934 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:26.935 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:26.935 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:26.935 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:26.935 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:26.936 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:26.936 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:26.936 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:26.936 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T18:58:26.936 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:26.936 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:26.937 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:26.937 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T18:58:26.937 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:26.937 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:26.937 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:26.938 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:26.938 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:26.938 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:26.939 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:26.939 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:26.939 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:26.939 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:26.940 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:26.940 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:26.940 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:26.940 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:26.941 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:26.941 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:26.941 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:26.941 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:26.941 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:26.942 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:26.942 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:26.942 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:26.942 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:26.942 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:26.943 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:26.943 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:26.943 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:26.943 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:26.943 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:26.944 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:26.944 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:26.944 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:26.944 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:26.944 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:26.945 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:26.945 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:26.945 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:26.945 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:26.945 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:26.945 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:26.946 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:26.946 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:26.946 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:26.946 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:26.946 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:26.947 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:26.948 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.404891_123.1 2023-10-12T18:58:31.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:31.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:31.950 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:31.993 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:31.993 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:32.045 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:32.046 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:32.046 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:32.046 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:32.046 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:32.047 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:32.047 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:32.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:32.047 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:32.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:32.048 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:32.048 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:32.048 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:32.048 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T18:58:32.048 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:32.049 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:32.049 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:32.049 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T18:58:32.049 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:32.049 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:32.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:32.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:32.050 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:32.050 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:32.050 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:32.051 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:32.051 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:32.051 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:32.051 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:32.051 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:32.051 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:32.052 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:32.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:32.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:32.052 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:32.052 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:32.053 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:32.053 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:32.053 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:32.053 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:32.053 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:32.054 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:32.054 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:32.054 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:32.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:32.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:32.055 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:32.055 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:32.055 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:32.055 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:32.055 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:32.056 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:32.056 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:32.056 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:32.056 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:32.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:32.057 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:32.057 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:32.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:32.057 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:32.057 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:32.057 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:32.058 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:32.058 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:32.059 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.470944_133.1 2023-10-12T18:58:37.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:37.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:37.061 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:37.105 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:37.106 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:37.159 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:37.159 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:37.159 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:37.160 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:37.160 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:37.160 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:37.160 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:37.160 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:37.161 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:37.161 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:37.161 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:37.161 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:37.162 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:37.162 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T18:58:37.162 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:37.162 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:37.163 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:37.163 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T18:58:37.163 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:37.163 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:37.163 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:37.164 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:37.164 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:37.164 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:37.164 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:37.164 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:37.165 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:37.165 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:37.165 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:37.165 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:37.165 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:37.165 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:37.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:37.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:37.166 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:37.166 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:37.166 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:37.167 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:37.167 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:37.167 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:37.167 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:37.167 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:37.168 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:37.168 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:37.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:37.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:37.168 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:37.168 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:37.169 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:37.169 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:37.169 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:37.169 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:37.169 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:37.170 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:37.170 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:37.170 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:37.170 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:37.171 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:37.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:37.171 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:37.171 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:37.172 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:37.172 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:37.172 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:37.173 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.473047_134.1 2023-10-12T18:58:42.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:42.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:42.175 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:42.220 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:42.220 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:42.274 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T18:58:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T18:58:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:42.287 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:42.288 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.435183_128.1 2023-10-12T18:58:47.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:47.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:47.290 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:47.333 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:47.334 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:47.388 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:47.388 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:47.389 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:47.389 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:47.389 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:47.389 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:47.390 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:47.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:47.390 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:47.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:47.390 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:47.391 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:47.391 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:47.391 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:58:47.391 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:47.391 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:47.392 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:47.392 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:58:47.392 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:47.392 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:47.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:47.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:47.393 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:47.393 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:47.393 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:47.393 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:47.393 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:47.394 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:47.394 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:47.394 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:47.394 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:47.394 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:47.395 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:47.395 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:47.395 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:47.395 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:47.395 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:47.396 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:47.396 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:47.396 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:47.396 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:47.396 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:47.397 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:47.397 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:47.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:47.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:47.397 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:47.398 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:47.398 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:47.398 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:47.398 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:47.398 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:47.399 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:47.399 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:47.399 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:47.399 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:47.399 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:47.400 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:47.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:47.400 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:47.400 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:47.400 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:47.401 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:47.401 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:47.402 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:58:52.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:52.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:52.404 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:52.447 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:52.447 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:52.497 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:52.498 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:52.498 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:52.498 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:52.498 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:52.499 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:52.499 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:52.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:52.499 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:52.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:52.500 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:52.500 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:52.500 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:52.500 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T18:58:52.500 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:52.501 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:52.501 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:52.501 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T18:58:52.501 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:52.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:52.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:52.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:52.502 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:52.502 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:52.502 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:52.503 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:52.503 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:52.503 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:52.503 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:52.503 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:52.504 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:52.504 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:52.504 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:52.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:52.504 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:52.504 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:52.505 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:52.505 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:52.505 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:52.505 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:52.505 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:52.506 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:52.506 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:52.506 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:52.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:52.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:52.506 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:52.507 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:52.507 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:52.507 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:52.507 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:52.507 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:52.508 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:52.508 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:52.508 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:52.508 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:52.508 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:52.509 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:52.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:52.509 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:52.509 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:52.509 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:52.509 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:52.510 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:52.511 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.408247_125.1 2023-10-12T18:58:57.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:57.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:58:57.513 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:58:57.556 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:58:57.556 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:58:57.607 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:58:57.608 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:58:57.608 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:58:57.608 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:58:57.608 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:58:57.609 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:58:57.609 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:57.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:57.609 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:58:57.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:57.610 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:58:57.610 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:57.610 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:57.610 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T18:58:57.611 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:57.611 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:58:57.611 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:57.611 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T18:58:57.611 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:57.612 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:57.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:57.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:57.612 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:58:57.612 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:57.613 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:57.613 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:58:57.613 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:57.613 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:57.613 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:57.614 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:58:57.614 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:57.614 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:57.614 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:57.614 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:57.615 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:58:57.615 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:57.615 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:57.615 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:57.615 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:57.616 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:57.616 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:57.616 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:57.616 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:57.616 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:57.617 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:57.617 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:58:57.617 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:58:57.617 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:58:57.617 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:58:57.618 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:58:57.618 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:58:57.618 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:58:57.618 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:58:57.618 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:58:57.619 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:58:57.619 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:57.619 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:57.619 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:58:57.619 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:58:57.620 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:58:57.620 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:58:57.620 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:58:57.620 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:58:57.620 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:58:57.621 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.435183_128.1 2023-10-12T18:59:02.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:02.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:02.624 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:02.667 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:02.667 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:02.719 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:02.719 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:02.719 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:02.720 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:02.720 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:02.720 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:02.720 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:02.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:02.721 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:02.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:02.721 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:02.721 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:02.721 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:02.722 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T18:59:02.722 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:02.722 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:02.722 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:02.722 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T18:59:02.723 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:02.723 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:02.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:02.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:02.723 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:02.724 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:02.724 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:02.724 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:02.724 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:02.724 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:02.724 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:02.725 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:02.725 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:02.725 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:02.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:02.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:02.726 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:02.726 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:02.726 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:02.727 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:02.727 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:02.727 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:02.727 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:02.727 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:02.727 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:02.728 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:02.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:02.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:02.728 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:02.728 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:02.729 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:02.729 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:02.729 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:02.729 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:02.729 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:02.730 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:02.730 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:02.730 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:02.730 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:02.730 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:02.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:02.731 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:02.731 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:02.731 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:02.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:02.731 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:02.733 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.436071_129.1 2023-10-12T18:59:07.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:07.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:07.736 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:07.782 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:07.783 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:07.837 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T18:59:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T18:59:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:07.846 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:07.846 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:07.847 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:07.847 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:07.847 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:07.850 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:07.850 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:07.850 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:07.850 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:07.851 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.404891_123.1 2023-10-12T18:59:12.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:12.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:12.854 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:12.896 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:12.897 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:12.978 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:12.978 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:12.978 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:12.979 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:12.979 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:12.979 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:12.979 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:12.979 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:12.980 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:12.980 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:12.980 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:12.980 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:12.980 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:12.981 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:59:12.981 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:12.981 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:12.981 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:12.981 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:59:12.982 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:12.982 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:12.982 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:12.982 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:12.982 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:12.983 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:12.983 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:12.983 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:12.983 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:12.983 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:12.983 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:12.984 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:12.984 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:12.984 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:12.984 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:12.984 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:12.985 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:12.985 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:12.985 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:12.985 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:12.985 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:12.985 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:12.986 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:12.986 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:12.986 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:12.986 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:12.986 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:12.987 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:12.987 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:12.987 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:12.987 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:12.987 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:12.988 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:12.988 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:12.988 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:12.988 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:12.988 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:12.988 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:12.989 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:12.989 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:12.989 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:12.989 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:12.989 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:12.990 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:12.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:12.990 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:12.991 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:59:17.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:17.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:17.993 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:18.070 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:18.071 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:18.408 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:18.409 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:18.409 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:18.409 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:18.409 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:18.409 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:18.410 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:18.410 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:18.410 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:18.410 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:18.410 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.586855_149.1", 2023-10-12T18:59:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.586855Z", 2023-10-12T18:59:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:18.421 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:18.422 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.586855_149.1 2023-10-12T18:59:23.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:23.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:23.424 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:23.512 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:23.513 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:23.601 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:23.601 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:23.601 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:23.602 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:23.602 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:23.602 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:23.602 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:23.602 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:23.603 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:23.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:23.603 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:23.603 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:23.603 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:23.603 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T18:59:23.604 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:23.604 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:23.604 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:23.604 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T18:59:23.604 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:23.605 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:23.605 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:23.605 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:23.605 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:23.605 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:23.606 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:23.606 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:23.606 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:23.606 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:23.606 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:23.606 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:23.607 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:23.607 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:23.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:23.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:23.607 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:23.608 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:23.608 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:23.608 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:23.608 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:23.608 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:23.609 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:23.609 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:23.609 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:23.609 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:23.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:23.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:23.610 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:23.610 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:23.610 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:23.610 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:23.610 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:23.611 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:23.611 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:23.611 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:23.611 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:23.611 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:23.611 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:23.612 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:23.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:23.612 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:23.612 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:23.612 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:23.613 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:23.613 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:23.614 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.392700_120.1 2023-10-12T18:59:28.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:28.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:28.616 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:28.664 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:28.664 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:28.748 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:28.748 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:28.749 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:28.749 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:28.749 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:28.749 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:28.749 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:28.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:28.750 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:28.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:28.750 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:28.750 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:28.751 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:28.751 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T18:59:28.751 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:28.751 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:28.752 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:28.752 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T18:59:28.752 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:28.752 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:28.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:28.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:28.753 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:28.753 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:28.753 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:28.753 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:28.754 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:28.754 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:28.754 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:28.755 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:28.755 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:28.755 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:28.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:28.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:28.756 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:28.756 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:28.756 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:28.756 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:28.756 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:28.757 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:28.757 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:28.757 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:28.757 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:28.757 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:28.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:28.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:28.758 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:28.758 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:28.759 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:28.759 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:28.759 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:28.759 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:28.760 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:28.760 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:28.760 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:28.760 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:28.760 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:28.761 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:28.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:28.761 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:28.761 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:28.761 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:28.762 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:28.762 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:28.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.404891_123.1 2023-10-12T18:59:33.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:33.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:33.765 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:33.864 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:33.864 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:34.041 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:34.042 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:34.042 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:34.042 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:34.042 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:34.043 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:34.043 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:34.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:34.043 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:34.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:34.044 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:34.044 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:34.044 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:34.044 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T18:59:34.044 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:34.045 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:34.045 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:34.045 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T18:59:34.045 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:34.045 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:34.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:34.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:34.046 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:34.046 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:34.046 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:34.047 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:34.047 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:34.047 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:34.047 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:34.047 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:34.047 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:34.048 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:34.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:34.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:34.048 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:34.048 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:34.049 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:34.049 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:34.049 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:34.049 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:34.049 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:34.050 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:34.050 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:34.050 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:34.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:34.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:34.051 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:34.051 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:34.051 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:34.051 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:34.051 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:34.051 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:34.052 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:34.052 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:34.052 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:34.052 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:34.052 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:34.053 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:34.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:34.053 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:34.053 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:34.053 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:34.054 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:34.054 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:34.055 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.408247_125.1 2023-10-12T18:59:39.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:39.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:39.058 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:39.101 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:39.101 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:39.153 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:39.154 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:39.154 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:39.154 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:39.155 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:39.155 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:39.155 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:39.156 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:39.156 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:39.156 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:39.156 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:39.157 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:39.157 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:39.157 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T18:59:39.157 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:39.157 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:39.158 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:39.158 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T18:59:39.158 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:39.158 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:39.158 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:39.159 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:39.159 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:39.159 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:39.159 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:39.159 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:39.160 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:39.160 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:39.160 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:39.160 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:39.161 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:39.161 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:39.161 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:39.161 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:39.161 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:39.162 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:39.162 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:39.162 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:39.162 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:39.163 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:39.163 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:39.163 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:39.163 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:39.163 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:39.163 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:39.164 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:39.164 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:39.164 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:39.164 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:39.164 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:39.165 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:39.165 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:39.165 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:39.165 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:39.165 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:39.165 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:39.166 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:39.166 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:39.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:39.166 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:39.166 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:39.167 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:39.167 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:39.167 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:39.168 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.473047_134.1 2023-10-12T18:59:44.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:44.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:44.171 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:44.233 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:44.233 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:44.363 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:44.363 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:44.363 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:44.364 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:44.364 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:44.364 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:44.364 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:44.364 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:44.365 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:44.365 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:44.365 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:44.365 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:44.366 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:44.366 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:59:44.366 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:44.366 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:44.366 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:44.367 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:59:44.367 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:44.367 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:44.367 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:44.367 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:44.368 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:44.368 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:44.368 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:44.368 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:44.368 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:44.368 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:44.369 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:44.369 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:44.369 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:44.369 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:44.369 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:44.370 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:44.370 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:44.370 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:44.370 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:44.370 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:44.371 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:44.371 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:44.371 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:44.371 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:44.371 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:44.372 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:44.372 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:44.372 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:44.372 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:44.372 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:44.373 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:44.373 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:44.373 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:44.373 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:44.373 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:44.373 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:44.374 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:44.374 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:44.374 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:44.374 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:44.374 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:44.375 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:44.375 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:44.375 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:44.375 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:44.375 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:44.376 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:59:49.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:49.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:49.378 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:49.467 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:49.467 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:49.678 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:49.678 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:49.679 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:49.679 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:49.679 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:49.679 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:49.680 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:49.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:49.680 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:49.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:49.680 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:49.681 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:49.681 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:49.681 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:59:49.681 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:49.682 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:49.682 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:49.682 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:59:49.682 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:49.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:49.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:49.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:49.683 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:49.683 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:49.684 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:49.684 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:49.684 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:49.684 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:49.684 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:49.685 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:49.685 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:49.685 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:49.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:49.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:49.686 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:49.686 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:49.686 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:49.686 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:49.687 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:49.687 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:49.687 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:49.687 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:49.687 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:49.688 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:49.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:49.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:49.688 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:49.688 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:49.689 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:49.689 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:49.689 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:49.689 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:49.690 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:49.690 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:49.690 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:49.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:49.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:49.691 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:49.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:49.691 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:49.691 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:49.692 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:49.692 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:49.692 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:49.693 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:59:54.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:54.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:54.696 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:54.776 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:54.776 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:54.831 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:54.831 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:54.831 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:54.831 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:54.831 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:54.832 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:54.832 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:54.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:54.832 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:54.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:54.833 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:54.833 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:54.833 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:54.833 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:59:54.833 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:54.834 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:54.834 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:54.834 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:59:54.834 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:54.834 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:54.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:54.835 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:54.835 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:54.835 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:54.835 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:54.836 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:54.836 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:54.836 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:54.836 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:54.836 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:54.837 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:54.837 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:54.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:54.837 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:54.837 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:54.837 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:54.838 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:54.838 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:54.838 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:54.838 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:54.838 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:54.839 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:54.839 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:54.839 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:54.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:54.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:54.840 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:54.840 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:54.840 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:54.840 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:54.840 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:54.840 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:54.841 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:54.841 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:54.841 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:54.841 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:54.841 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:54.842 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:54.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:54.842 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:54.842 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:54.842 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:54.843 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:54.843 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:54.844 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T18:59:59.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:59.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T18:59:59.846 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T18:59:59.902 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T18:59:59.902 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T18:59:59.951 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T18:59:59.951 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T18:59:59.952 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T18:59:59.952 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T18:59:59.952 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T18:59:59.953 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T18:59:59.953 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:59.953 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:59.953 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T18:59:59.954 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:59.954 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T18:59:59.954 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:59.955 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:59.955 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T18:59:59.955 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:59.955 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T18:59:59.955 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:59.956 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T18:59:59.956 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:59.956 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:59.956 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:59.956 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:59.956 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T18:59:59.957 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:59.957 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:59.957 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T18:59:59.957 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:59.957 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:59.958 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:59.958 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T18:59:59.958 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:59.958 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:59.959 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:59.959 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:59.959 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T18:59:59.959 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:59.959 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:59.960 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:59.960 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:59.960 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:59.960 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:59.960 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:59.961 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:59.961 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:59.961 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:59.961 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T18:59:59.962 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T18:59:59.962 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T18:59:59.962 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T18:59:59.962 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T18:59:59.962 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T18:59:59.963 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T18:59:59.963 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T18:59:59.963 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T18:59:59.963 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T18:59:59.963 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:59.963 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:59.964 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T18:59:59.964 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T18:59:59.964 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T18:59:59.964 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T18:59:59.964 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T18:59:59.965 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T18:59:59.965 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T18:59:59.966 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T19:00:04.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:04.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:04.969 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:05.013 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:05.013 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:05.062 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:05.063 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:05.063 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:05.063 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:05.063 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:05.065 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:05.065 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:05.065 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:05.065 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:00:05.066 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:05.066 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:05.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:05.066 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:00:05.066 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:05.067 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:05.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:05.067 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:05.067 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:05.067 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:05.068 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:05.068 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:05.068 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:05.068 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:05.068 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:05.069 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:05.069 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:05.069 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:05.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:05.069 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:05.070 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:05.070 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:05.070 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:05.070 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:05.070 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:05.071 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:05.071 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:05.071 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:05.071 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:05.071 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:05.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:05.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:05.072 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:05.072 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:05.072 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:05.073 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:05.073 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:05.073 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:05.073 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:05.073 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:05.073 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:05.074 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:05.074 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:05.074 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:05.074 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:05.074 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:05.075 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:05.075 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:05.075 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:05.075 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:05.076 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.470944_133.1 2023-10-12T19:00:10.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:10.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:10.078 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:10.121 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:10.122 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:10.171 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:10.172 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:10.172 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:10.172 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:10.172 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:10.173 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:10.173 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:10.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:10.173 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:10.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:10.174 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:10.174 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:10.174 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:10.174 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:00:10.175 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:10.175 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:10.175 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:10.175 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:00:10.176 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:10.176 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:10.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:10.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:10.176 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:10.177 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:10.177 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:10.177 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:10.178 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:10.178 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:10.178 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:10.178 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:10.179 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:10.179 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:10.179 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:10.179 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:10.180 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:10.180 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:10.180 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:10.180 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:10.180 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:10.180 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:10.181 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:10.181 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:10.181 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:10.181 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:10.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:10.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:10.182 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:10.182 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:10.182 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:10.182 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:10.183 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:10.183 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:10.183 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:10.183 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:10.183 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:10.184 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:10.184 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:10.184 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:10.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:10.184 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:10.185 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:10.185 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:10.185 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:10.185 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:10.186 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.408247_125.1 2023-10-12T19:00:15.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:15.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:15.189 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:15.234 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:15.235 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:15.286 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:15.287 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:15.287 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:15.287 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:15.288 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:15.288 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:15.288 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:15.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:15.288 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:15.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:15.289 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:15.289 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:15.289 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:15.290 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:00:15.290 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:15.290 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:15.290 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:15.290 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:00:15.291 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:15.291 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:15.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:15.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:15.291 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:15.292 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:15.292 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:15.292 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:15.292 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:15.292 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:15.292 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:15.293 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:15.293 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:15.293 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:15.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:15.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:15.294 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:15.294 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:15.294 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:15.294 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:15.294 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:15.295 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:15.295 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:15.295 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:15.295 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:15.295 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:15.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:15.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:15.296 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:15.296 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:15.296 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:15.296 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:15.297 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:15.297 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:15.297 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:15.297 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:15.297 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:15.298 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:15.298 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:15.298 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:15.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:15.298 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:15.299 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:15.299 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:15.299 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:15.299 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:15.300 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.392700_120.1 2023-10-12T19:00:20.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:20.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:20.303 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:20.378 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:20.379 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:20.428 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:20.429 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:20.429 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:20.429 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:20.429 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:20.429 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:20.429 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:20.430 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:20.430 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:20.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:20.430 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:00:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:20.432 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:20.432 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:00:20.432 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:20.432 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:20.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:20.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:20.433 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:20.433 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:20.434 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:20.434 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:20.434 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:20.434 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:20.436 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:20.436 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:20.436 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:20.436 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:20.437 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:20.437 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:20.437 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:20.437 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:20.438 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:20.438 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:20.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:20.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:20.438 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:20.439 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:20.439 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:20.439 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:20.439 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:20.439 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:20.440 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:20.440 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:20.440 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:20.440 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:20.440 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:20.440 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:20.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:20.441 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:20.441 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:20.441 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:20.441 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:20.442 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:20.443 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.407338_124.1 2023-10-12T19:00:25.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:25.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:25.445 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:25.490 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:25.490 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:25.539 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:25.539 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:25.540 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:25.540 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:25.540 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:25.540 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:25.541 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:25.541 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:25.541 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:25.541 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:25.541 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:25.542 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:25.542 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:25.542 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:00:25.542 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:25.542 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:25.543 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:25.543 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:00:25.543 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:25.543 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:25.543 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:25.544 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:25.544 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:25.544 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:25.544 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:25.544 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:25.545 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:25.545 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:25.545 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:25.545 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:25.545 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:25.545 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:25.546 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:25.546 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:25.546 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:25.546 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:25.547 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:25.547 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:25.547 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:25.547 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:25.548 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:25.548 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:25.548 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:25.548 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:25.548 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:25.549 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:25.549 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:25.549 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:25.549 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:25.549 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:25.550 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:25.550 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:25.550 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:25.550 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:25.551 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:25.551 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:25.551 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:25.551 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:25.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:25.552 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:25.552 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:25.552 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:25.552 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:25.552 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:25.553 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.392700_120.1 2023-10-12T19:00:30.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:30.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:30.555 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:30.647 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:30.647 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:30.725 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:30.726 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:30.726 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:30.726 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:30.726 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:30.726 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:30.727 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:30.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:30.727 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:30.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:30.727 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:30.728 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:30.728 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:30.728 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:00:30.728 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:30.728 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:30.729 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:30.729 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:00:30.729 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:30.729 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:30.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:30.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:30.730 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:30.730 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:30.730 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:30.730 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:30.731 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:30.731 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:30.731 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:30.731 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:30.731 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:30.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:30.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:30.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:30.732 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:30.732 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:30.732 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:30.733 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:30.733 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:30.733 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:30.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:30.733 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:30.734 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:30.734 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:30.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:30.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:30.734 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:30.734 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:30.735 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:30.735 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:30.735 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:30.735 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:30.735 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:30.736 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:30.736 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:30.736 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:30.736 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:30.736 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:30.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:30.737 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:30.737 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:30.737 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:30.737 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:30.737 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:30.738 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T19:00:35.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:35.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:35.741 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:35.828 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:35.828 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:35.905 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:35.906 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:35.906 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:35.906 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:35.906 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:35.906 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:35.907 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:35.907 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:35.907 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:35.907 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:35.907 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:35.908 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:35.908 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:35.908 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.574307_146.1", 2023-10-12T19:00:35.908 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:35.908 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:35.909 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:35.909 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.574307Z", 2023-10-12T19:00:35.909 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:35.909 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:35.909 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:35.910 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:35.910 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:35.910 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:35.910 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:35.910 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:35.911 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:35.911 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:35.911 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:35.911 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:35.911 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:35.912 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:35.912 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:35.912 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:35.912 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:35.912 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:35.912 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:35.913 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:35.913 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:35.913 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:35.913 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:35.913 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:35.914 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:35.914 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:35.914 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:35.914 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:35.914 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:35.915 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:35.915 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:35.915 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:35.915 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:35.916 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:35.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:35.916 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:35.916 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:35.916 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:35.917 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:35.917 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:35.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:35.917 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:35.917 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:35.918 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:35.918 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:35.918 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:35.919 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.574307_146.1 2023-10-12T19:00:40.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:40.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:40.921 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:40.997 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:40.997 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:41.046 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:41.046 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:00:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:00:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:41.059 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:41.060 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.392700_120.1 2023-10-12T19:00:46.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:46.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:46.063 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:46.124 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:46.125 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:46.175 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:46.175 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:46.176 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:46.176 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:46.176 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:46.176 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:46.176 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:46.177 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:46.177 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:46.177 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:46.177 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:46.177 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:46.178 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:46.178 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:00:46.178 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:46.178 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:46.178 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:46.179 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:00:46.179 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:46.179 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:46.179 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:46.179 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:46.180 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:46.180 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:46.180 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:46.180 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:46.180 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:46.180 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:46.181 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:46.181 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:46.181 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:46.181 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:46.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:46.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:46.182 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:46.182 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:46.182 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:46.182 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:46.183 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:46.183 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:46.183 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:46.183 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:46.183 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:46.184 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:46.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:46.184 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:46.184 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:46.184 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:46.185 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:46.185 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:46.185 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:46.185 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:46.185 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:46.187 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:46.187 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:46.187 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:46.187 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:46.188 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:46.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:46.188 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:46.188 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:46.188 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:46.189 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:46.189 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:46.190 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.436071_129.1 2023-10-12T19:00:51.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:51.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:51.193 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:51.236 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:51.237 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:51.285 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:51.285 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:51.286 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:51.286 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:51.286 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:51.286 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:51.287 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:51.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:51.287 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:51.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:51.287 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:51.288 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:51.288 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:51.288 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:00:51.288 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:51.288 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:51.289 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:51.289 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:00:51.289 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:51.289 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:51.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:51.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:51.290 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:51.290 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:51.290 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:51.291 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:51.291 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:51.291 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:51.291 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:51.291 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:51.292 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:51.292 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:51.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:51.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:51.292 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:51.293 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:51.293 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:51.293 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:51.293 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:51.293 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:51.294 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:51.294 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:51.294 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:51.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:51.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:51.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:51.295 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:51.295 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:51.295 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:51.295 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:51.295 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:51.296 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:51.296 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:51.296 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:51.296 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:51.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:51.297 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:51.297 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:51.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:51.297 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:51.297 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:51.298 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:51.298 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:51.298 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:51.299 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.396100_122.1 2023-10-12T19:00:56.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:56.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:00:56.302 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:00:56.343 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:00:56.344 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:00:56.393 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:00:56.393 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:00:56.393 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:00:56.394 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:00:56.394 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:00:56.394 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:00:56.394 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:56.395 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:56.395 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:00:56.395 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:56.395 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:00:56.395 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:56.395 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:56.396 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:00:56.396 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:56.396 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:00:56.396 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:56.396 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:00:56.397 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:56.397 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:56.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:56.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:56.397 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:00:56.398 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:56.398 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:56.398 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:00:56.398 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:56.398 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:56.399 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:56.399 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:00:56.399 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:56.399 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:56.399 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:56.400 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:56.400 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:00:56.400 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:56.400 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:56.400 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:56.400 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:56.401 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:56.401 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:56.401 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:56.401 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:56.401 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:56.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:56.402 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:00:56.402 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:00:56.402 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:00:56.402 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:00:56.403 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:00:56.403 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:00:56.403 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:00:56.403 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:00:56.403 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:00:56.404 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:00:56.404 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:56.404 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:56.404 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:00:56.404 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:00:56.405 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:00:56.405 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:00:56.405 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:00:56.405 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:00:56.405 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:00:56.406 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.436071_129.1 2023-10-12T19:01:01.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:01.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:01.409 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:01.450 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:01.450 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:01.500 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:01.500 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:01.501 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:01.501 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:01.501 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:01.501 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:01.502 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:01.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:01.502 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:01.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:01.502 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:01.503 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:01.503 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:01.503 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.764432_172.1", 2023-10-12T19:01:01.503 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:01.503 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:01.504 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:01.504 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.764432Z", 2023-10-12T19:01:01.504 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:01.504 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:01.504 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:01.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:01.505 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:01.505 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:01.505 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:01.505 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:01.505 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:01.506 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:01.506 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:01.506 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:01.507 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:01.507 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:01.507 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:01.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:01.507 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:01.507 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:01.508 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:01.508 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:01.508 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:01.508 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:01.509 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:01.509 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:01.509 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:01.509 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:01.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:01.510 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:01.510 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:01.510 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:01.510 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:01.510 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:01.511 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:01.511 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:01.511 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:01.511 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:01.511 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:01.512 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:01.512 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:01.512 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:01.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:01.512 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:01.512 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:01.513 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:01.513 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:01.513 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:01.514 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.764432_172.1 2023-10-12T19:01:06.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:06.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:06.516 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:06.590 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:06.590 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:06.642 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:06.642 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:06.642 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:06.642 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:06.643 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:06.643 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:06.643 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:06.643 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:06.643 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:06.644 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:06.644 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:06.644 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:06.644 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:06.644 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:01:06.644 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:06.645 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:06.645 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:06.645 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:01:06.645 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:06.645 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:06.646 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:06.646 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:06.646 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:06.646 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:06.646 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:06.646 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:06.647 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:06.647 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:06.647 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:06.647 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:06.647 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:06.648 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:06.648 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:06.648 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:06.648 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:06.648 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:06.649 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:06.649 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:06.649 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:06.649 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:06.649 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:06.650 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:06.650 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:06.650 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:06.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:06.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:06.651 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:06.651 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:06.651 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:06.651 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:06.651 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:06.651 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:06.652 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:06.652 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:06.652 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:06.652 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:06.652 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:06.653 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:06.653 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:06.653 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:06.653 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:06.653 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:06.654 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:06.654 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:06.655 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.407338_124.1 2023-10-12T19:01:11.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:11.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:11.657 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:11.699 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:11.700 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:11.749 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:11.749 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:11.749 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:11.750 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:11.750 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:11.750 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:11.750 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:11.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:11.751 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:11.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:11.751 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:11.751 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:11.752 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:11.752 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.568332_143.1", 2023-10-12T19:01:11.752 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:11.752 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:11.752 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:11.753 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.568332Z", 2023-10-12T19:01:11.753 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:11.753 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:11.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:11.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:11.754 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:11.754 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:11.754 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:11.755 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:11.755 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:11.755 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:11.755 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:11.755 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:11.756 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:11.757 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:11.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:11.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:11.757 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:11.757 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:11.758 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:11.758 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:11.758 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:11.758 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:11.759 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:11.759 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:11.759 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:11.759 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:11.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:11.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:11.760 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:11.760 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:11.760 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:11.760 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:11.760 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:11.761 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:11.761 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:11.761 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:11.761 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:11.761 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:11.762 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:11.762 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:11.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:11.762 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:11.763 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:11.763 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:11.763 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:11.763 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:11.765 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.568332_143.1 2023-10-12T19:01:16.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:16.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:16.767 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:16.809 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:16.809 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:16.870 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:16.870 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:16.870 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:16.871 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:16.871 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:16.871 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:16.871 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:16.871 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:16.872 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:16.872 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:16.872 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:16.872 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:16.872 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:16.873 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:01:16.873 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:16.873 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:16.873 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:16.873 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:01:16.874 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:16.874 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:16.874 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:16.874 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:16.874 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:16.875 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:16.875 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:16.875 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:16.875 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:16.875 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:16.876 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:16.876 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:16.876 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:16.876 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:16.876 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:16.876 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:16.877 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:16.877 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:16.877 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:16.877 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:16.877 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:16.878 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:16.878 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:16.878 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:16.878 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:16.878 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:16.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:16.879 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:16.879 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:16.879 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:16.879 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:16.880 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:16.880 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:16.880 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:16.880 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:16.880 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:16.880 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:16.881 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:16.881 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:16.881 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:16.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:16.881 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:16.882 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:16.882 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:16.882 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:16.882 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:16.883 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.407338_124.1 2023-10-12T19:01:21.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:21.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:21.886 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:21.928 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:21.928 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:21.978 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:21.978 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:21.979 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:21.979 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:21.979 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:21.979 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:21.979 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:21.980 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:21.980 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:21.980 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:21.980 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:21.980 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:21.980 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:21.981 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:01:21.981 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:21.981 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:21.981 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:21.981 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:01:21.982 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:21.982 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:21.982 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:21.982 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:21.982 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:21.983 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:21.983 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:21.983 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:21.983 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:21.983 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:21.984 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:21.984 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:21.984 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:21.984 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:21.984 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:21.985 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:21.985 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:21.985 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:21.985 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:21.985 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:21.985 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:21.986 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:21.986 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:21.986 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:21.986 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:21.986 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:21.987 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:21.987 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:21.987 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:21.987 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:21.987 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:21.988 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:21.988 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:21.988 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:21.988 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:21.988 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:21.988 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:21.989 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:21.989 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:21.989 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:21.989 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:21.989 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:21.990 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:21.990 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:21.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:21.990 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:21.991 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.436071_129.1 2023-10-12T19:01:26.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:26.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:26.994 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:27.123 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:27.123 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:27.174 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:27.174 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:27.175 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:27.175 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:27.175 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:27.176 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:27.176 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:27.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:27.176 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:27.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:27.177 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:27.177 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:27.177 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:27.177 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:01:27.177 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:27.177 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:27.178 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:27.178 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:01:27.178 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:27.179 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:27.179 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:27.179 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:27.180 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:27.180 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:27.181 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:27.181 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:27.181 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:27.182 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:27.182 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:27.183 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:27.183 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:27.183 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:27.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:27.184 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:27.184 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:27.185 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:27.185 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:27.185 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:27.186 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:27.186 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:27.186 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:27.187 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:27.187 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:27.187 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:27.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:27.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:27.188 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:27.189 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:27.189 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:27.189 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:27.190 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:27.190 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:27.190 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:27.191 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:27.191 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:27.191 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:27.192 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:27.192 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:27.192 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:27.192 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:27.193 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:27.193 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:27.193 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:27.193 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:27.194 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T19:01:32.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:32.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:32.197 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:32.995 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:32.995 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:33.056 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:33.057 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:33.057 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:33.057 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:33.057 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:33.057 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:33.057 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:33.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:33.058 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:33.058 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:33.058 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:33.059 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:33.059 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:33.059 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:01:33.059 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:33.059 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:33.059 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:33.060 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:01:33.060 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:33.060 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:33.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:33.060 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:33.061 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:33.061 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:33.061 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:33.061 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:33.061 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:33.062 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:33.062 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:33.062 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:33.062 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:33.062 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:33.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:33.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:33.063 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:33.063 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:33.063 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:33.063 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:33.064 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:33.065 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:33.066 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:33.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:33.068 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:33.068 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:33.068 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:33.068 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:33.068 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:33.069 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.408247_125.1 2023-10-12T19:01:38.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:38.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:38.072 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:38.116 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:38.116 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:38.171 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:38.171 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:38.172 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:38.172 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:38.172 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:38.172 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:38.172 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:38.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:38.173 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:38.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:38.174 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:38.174 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:38.174 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:38.174 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:01:38.174 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:38.175 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:38.175 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:38.175 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:01:38.175 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:38.175 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:38.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:38.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:38.176 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:38.176 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:38.176 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:38.177 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:38.177 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:38.177 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:38.177 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:38.177 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:38.178 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:38.178 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:38.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:38.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:38.178 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:38.179 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:38.179 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:38.179 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:38.179 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:38.179 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:38.180 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:38.180 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:38.180 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:38.180 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:38.180 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:38.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:38.181 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:38.181 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:38.181 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:38.181 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:38.181 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:38.182 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:38.182 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:38.182 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:38.182 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:38.182 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:38.183 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:38.183 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:38.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:38.183 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:38.184 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:38.184 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:38.184 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:38.184 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:38.185 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T19:01:43.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:43.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:43.188 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:43.290 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:43.291 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:43.345 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:43.345 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:43.345 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:43.346 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:43.346 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:43.346 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:43.346 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:43.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:43.347 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:43.347 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:43.347 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:43.347 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:43.348 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:43.348 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:01:43.348 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:43.348 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:43.348 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:43.349 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:01:43.349 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:43.349 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:43.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:43.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:43.350 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:43.350 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:43.350 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:43.350 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:43.350 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:43.351 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:43.351 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:43.351 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:43.351 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:43.351 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:43.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:43.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:43.352 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:43.352 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:43.352 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:43.352 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:43.353 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:43.353 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:43.353 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:43.353 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:43.353 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:43.354 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:43.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:43.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:43.354 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:43.354 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:43.355 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:43.355 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:43.355 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:43.355 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:43.355 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:43.356 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:43.356 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:43.356 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:43.356 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:43.356 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:43.357 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:43.357 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:43.357 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:43.357 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:43.357 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:43.358 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:43.359 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.470944_133.1 2023-10-12T19:01:48.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:48.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:48.361 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:48.404 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:48.404 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:48.464 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:48.464 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:48.465 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:48.465 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:48.465 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:48.465 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:48.465 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:48.466 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:48.466 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:48.466 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:48.466 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:48.466 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:48.467 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:48.467 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:01:48.467 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:48.467 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:48.467 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:48.468 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:01:48.468 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:48.468 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:48.468 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:48.468 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:48.469 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:48.469 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:48.469 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:48.469 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:48.469 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:48.469 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:48.470 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:48.470 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:48.470 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:48.470 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:48.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:48.471 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:48.471 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:48.471 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:48.471 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:48.471 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:48.471 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:48.472 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:48.472 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:48.472 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:48.472 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:48.472 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:48.473 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:48.473 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:48.473 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:48.473 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:48.473 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:48.474 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:48.474 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:48.474 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:48.474 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:48.474 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:48.475 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:48.475 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:48.475 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:48.475 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:48.475 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:48.476 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:48.476 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:48.476 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:48.476 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:48.476 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:48.477 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.436071_129.1 2023-10-12T19:01:53.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:53.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:53.482 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:53.527 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:53.528 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:53.582 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:53.582 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:53.583 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:53.583 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:53.583 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:53.583 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:53.583 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:53.584 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:53.584 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:53.584 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:53.584 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:53.584 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:53.585 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:53.585 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:01:53.585 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:53.585 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:53.585 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:53.586 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:01:53.586 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:53.586 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:53.586 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:53.586 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:53.587 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:53.587 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:53.587 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:53.587 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:53.587 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:53.588 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:53.588 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:53.588 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:53.588 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:53.588 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:53.588 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:53.589 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:53.589 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:53.589 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:53.589 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:53.589 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:53.589 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:53.590 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:53.590 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:53.590 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:53.590 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:53.591 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:53.591 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:53.591 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:53.591 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:53.591 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:53.591 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:53.592 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:53.592 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:53.592 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:53.592 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:53.592 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:53.593 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:53.593 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:53.593 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:53.593 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:53.593 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:53.594 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:53.594 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:53.594 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:53.594 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:53.594 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:53.595 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T19:01:58.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:58.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:01:58.598 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:01:58.644 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:01:58.644 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:01:58.698 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:01:58.698 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:01:58.698 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:01:58.698 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:01:58.699 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:01:58.699 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:01:58.699 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:58.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:58.699 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:01:58.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:58.700 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:01:58.700 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:58.700 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:58.700 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:01:58.700 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:58.701 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:01:58.701 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:58.701 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:01:58.701 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:58.701 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:58.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:58.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:58.702 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:01:58.702 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:58.702 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:58.703 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:01:58.703 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:58.703 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:58.703 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:58.703 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:01:58.704 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:58.704 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:58.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:58.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:58.704 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:01:58.704 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:58.705 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:58.705 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:58.705 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:58.705 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:58.705 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:58.706 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:58.706 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:58.706 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:58.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:58.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:01:58.706 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:01:58.707 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:01:58.707 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:01:58.707 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:01:58.707 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:01:58.707 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:01:58.708 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:01:58.708 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:01:58.708 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:01:58.708 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:58.708 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:58.709 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:01:58.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:01:58.709 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:01:58.709 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:01:58.709 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:01:58.709 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:01:58.710 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:01:58.711 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.396100_122.1 2023-10-12T19:02:03.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:03.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:03.714 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:03.757 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:03.758 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:03.809 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:03.810 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:03.810 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:03.810 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:03.810 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:03.811 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:03.811 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:03.811 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:03.811 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:03.811 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:03.812 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:03.812 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:03.812 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:03.812 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:02:03.813 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:03.813 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:03.813 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:03.813 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:02:03.813 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:03.814 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:03.814 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:03.814 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:03.814 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:03.815 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:03.815 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:03.815 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:03.815 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:03.816 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:03.816 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:03.816 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:03.816 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:03.816 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:03.817 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:03.817 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:03.817 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:03.818 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:03.818 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:03.818 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:03.818 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:03.818 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:03.819 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:03.819 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:03.819 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:03.819 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:03.820 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:03.820 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:03.820 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:03.820 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:03.820 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:03.821 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:03.821 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:03.821 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:03.821 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:03.822 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:03.822 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:03.822 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:03.822 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:03.823 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:03.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:03.823 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:03.823 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:03.823 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:03.824 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:03.824 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:03.825 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.436071_129.1 2023-10-12T19:02:08.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:08.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:08.827 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:08.869 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:08.870 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:08.921 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:08.921 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:08.921 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:08.921 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:08.922 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:08.922 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:08.922 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:08.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:08.922 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:08.923 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:08.923 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:08.923 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:08.923 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:08.923 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:02:08.924 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:08.924 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:08.924 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:08.924 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:02:08.924 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:08.925 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:08.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:08.925 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:08.925 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:08.925 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:08.926 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:08.926 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:08.926 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:08.926 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:08.927 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:08.927 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:08.927 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:08.927 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:08.927 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:08.928 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:08.928 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:08.928 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:08.928 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:08.928 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:08.928 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:08.929 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:08.929 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:08.929 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:08.929 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:08.930 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:08.930 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:08.930 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:08.930 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:08.930 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:08.931 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:08.931 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:08.931 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:08.931 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:08.931 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:08.931 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:08.932 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:08.932 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:08.932 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:08.932 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:08.932 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:08.933 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:08.933 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:08.933 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:08.933 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:08.933 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:08.934 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.392700_120.1 2023-10-12T19:02:13.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:13.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:13.937 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:13.983 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:13.984 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:14.037 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:14.037 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:14.037 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:14.038 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:14.038 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:14.038 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:14.038 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:14.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:14.039 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:14.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:14.039 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:14.040 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:14.040 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:14.040 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:02:14.040 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:14.040 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:14.041 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:14.041 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:02:14.041 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:14.041 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:14.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:14.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:14.042 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:14.042 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:14.042 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:14.043 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:14.043 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:14.043 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:14.043 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:14.043 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:14.044 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:14.044 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:14.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:14.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:14.045 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:14.045 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:14.045 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:14.045 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:14.046 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:14.046 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:14.046 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:14.047 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:14.047 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:14.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:14.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:14.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:14.048 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:14.048 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:14.048 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:14.048 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:14.048 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:14.049 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:14.049 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:14.049 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:14.049 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:14.049 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:14.050 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:14.050 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:14.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:14.051 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:14.051 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:14.051 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:14.051 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:14.051 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:14.053 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.404891_123.1 2023-10-12T19:02:19.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:19.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:19.055 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:19.101 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:19.101 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:19.152 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:19.153 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:19.153 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:19.153 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:19.153 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:19.154 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:19.154 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:19.154 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:19.154 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:19.154 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:19.155 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:19.155 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:19.155 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:19.155 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:02:19.155 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:19.156 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:19.156 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:19.156 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:02:19.156 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:19.156 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:19.157 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:19.157 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:19.157 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:19.157 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:19.158 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:19.158 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:19.158 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:19.158 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:19.158 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:19.159 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:19.159 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:19.159 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:19.159 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:19.160 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:19.160 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:19.160 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:19.160 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:19.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:19.165 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:19.165 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:19.165 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:19.165 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:19.166 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:19.167 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T19:02:24.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:24.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:24.170 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:24.216 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:24.217 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:24.277 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:24.277 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:24.278 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:24.278 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:24.278 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:24.278 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:24.278 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:24.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:24.279 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:24.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:24.279 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:24.279 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:24.280 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:24.280 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:02:24.280 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:24.280 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:24.280 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:24.281 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:02:24.281 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:24.281 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:24.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:24.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:24.282 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:24.282 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:24.282 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:24.282 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:24.282 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:24.283 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:24.283 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:24.283 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:24.283 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:24.283 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:24.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:24.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:24.284 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:24.284 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:24.284 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:24.284 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:24.285 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:24.285 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:24.285 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:24.285 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:24.285 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:24.285 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:24.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:24.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:24.286 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:24.286 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:24.286 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:24.287 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:24.287 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:24.287 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:24.287 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:24.287 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:24.288 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:24.288 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:24.288 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:24.288 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:24.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:24.289 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:24.289 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:24.289 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:24.289 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:24.290 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:24.291 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.470944_133.1 2023-10-12T19:02:29.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:29.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:29.294 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:29.339 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:29.340 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:29.391 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:29.392 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:29.392 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:29.392 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:29.392 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:29.393 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:29.393 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:29.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:29.393 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:29.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:29.394 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:29.394 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:29.394 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:29.394 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:02:29.394 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:29.395 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:29.395 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:29.395 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:02:29.395 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:29.395 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:29.396 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:29.396 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:29.396 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:29.396 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:29.396 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:29.396 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:29.397 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:29.397 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:29.397 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:29.397 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:29.397 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:29.398 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:29.398 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:29.398 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:29.398 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:29.398 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:29.399 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:29.399 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:29.399 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:29.399 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:29.399 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:29.400 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:29.400 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:29.400 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:29.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:29.400 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:29.400 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:29.401 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:29.401 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:29.401 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:29.401 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:29.401 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:29.402 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:29.402 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:29.402 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:29.402 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:29.402 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:29.403 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:29.403 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:29.403 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:29.403 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:29.403 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:29.403 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:29.404 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:29.405 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.436071_129.1 2023-10-12T19:02:34.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:34.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:34.408 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:34.451 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:34.451 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:34.503 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:34.503 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:34.504 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:34.504 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:34.504 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:34.504 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:34.504 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:34.505 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:34.505 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:34.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:34.505 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:34.505 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:34.506 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:34.506 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.755208_170.1", 2023-10-12T19:02:34.506 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:34.506 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:34.506 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:34.507 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.755208Z", 2023-10-12T19:02:34.507 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:34.507 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:34.507 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:34.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:34.508 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:34.508 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:34.508 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:34.508 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:34.508 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:34.509 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:34.509 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:34.509 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:34.509 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:34.509 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:34.510 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:34.510 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:34.510 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:34.510 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:34.510 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:34.511 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:34.511 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:34.511 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:34.511 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:34.511 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:34.512 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:34.512 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:34.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:34.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:34.512 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:34.513 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:34.513 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:34.513 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:34.513 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:34.513 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:34.514 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:34.514 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:34.514 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:34.514 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:34.514 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:34.515 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:34.515 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:34.515 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:34.515 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:34.515 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:34.515 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:34.516 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:34.517 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.755208_170.1 2023-10-12T19:02:39.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:39.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:39.520 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:39.563 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:39.563 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:39.613 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:39.614 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:39.614 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:39.614 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:39.614 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:39.614 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:39.615 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:39.615 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:39.615 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:39.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:39.615 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:39.616 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:39.616 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:39.616 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:02:39.616 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:39.616 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:39.617 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:39.617 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:02:39.617 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:39.617 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:39.617 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:39.618 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:39.618 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:39.618 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:39.618 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:39.618 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:39.619 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:39.619 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:39.619 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:39.619 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:39.619 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:39.620 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:39.620 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:39.620 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:39.620 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:39.620 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:39.621 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:39.621 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:39.621 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:39.621 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:39.621 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:39.622 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:39.622 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:39.622 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:39.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:39.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:39.622 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:39.623 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:39.623 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:39.623 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:39.623 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:39.623 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:39.624 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:39.624 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:39.624 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:39.624 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:39.624 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:39.625 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:39.625 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:39.625 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:39.625 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:39.625 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:39.625 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:39.626 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:39.627 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.392700_120.1 2023-10-12T19:02:44.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:44.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:44.629 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:44.672 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:44.672 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:44.721 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:02:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:02:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:44.737 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:44.738 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T19:02:49.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:49.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:02:49.741 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:02:49.788 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:49.788 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:49.842 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:49.842 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:02:49.843 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:02:49.843 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:02:49.843 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:02:49.843 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:49.844 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:49.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:49.844 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:02:49.844 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:49.844 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:02:49.845 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:49.845 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:49.845 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:02:49.845 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:49.845 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:02:49.846 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:49.846 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:02:49.846 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:49.846 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:49.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:49.847 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:49.847 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:02:49.847 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:49.847 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:49.847 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:02:49.848 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:49.848 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:49.848 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:49.848 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:02:49.848 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:49.849 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:49.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:49.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:49.849 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:02:49.849 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:49.850 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:49.850 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:49.850 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:49.850 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:49.850 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:49.851 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:49.851 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:49.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:49.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:49.851 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:49.851 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:02:49.852 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:49.852 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:02:49.852 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:02:49.852 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:02:49.852 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:02:49.853 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:02:49.853 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:02:49.853 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:49.853 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:49.853 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:49.853 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:49.854 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:49.854 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:02:49.854 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:02:49.854 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:02:49.854 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:49.855 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:49.856 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137065.700268_210.1 target=1_1697136751.393673_121.1 2023-10-12T19:02:54.857 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... tasks.rgw_multi.tests.test_datalog_autotrim ... FAIL 2023-10-12T19:02:54.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:02:54.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:02:54.859 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T19:02:54.901 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:54.901 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:54.918 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:54.918 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T19:02:54.919 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a2", 2023-10-12T19:02:54.919 INFO:teuthology.orchestra.run.smithi170.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-12T19:02:54.919 INFO:teuthology.orchestra.run.smithi170.stdout: "control_pool": "a2.rgw.control", 2023-10-12T19:02:54.919 INFO:teuthology.orchestra.run.smithi170.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-12T19:02:54.919 INFO:teuthology.orchestra.run.smithi170.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-12T19:02:54.920 INFO:teuthology.orchestra.run.smithi170.stdout: "log_pool": "a2.rgw.log", 2023-10-12T19:02:54.920 INFO:teuthology.orchestra.run.smithi170.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-12T19:02:54.920 INFO:teuthology.orchestra.run.smithi170.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-12T19:02:54.920 INFO:teuthology.orchestra.run.smithi170.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-12T19:02:54.920 INFO:teuthology.orchestra.run.smithi170.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-12T19:02:54.921 INFO:teuthology.orchestra.run.smithi170.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-12T19:02:54.921 INFO:teuthology.orchestra.run.smithi170.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-12T19:02:54.921 INFO:teuthology.orchestra.run.smithi170.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-12T19:02:54.921 INFO:teuthology.orchestra.run.smithi170.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-12T19:02:54.921 INFO:teuthology.orchestra.run.smithi170.stdout: "otp_pool": "a2.rgw.otp", 2023-10-12T19:02:54.921 INFO:teuthology.orchestra.run.smithi170.stdout: "system_key": { 2023-10-12T19:02:54.922 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "29UBGG6NS65BV4RP", 2023-10-12T19:02:54.922 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "JTLzR0q78OtzvD8VcNOd7UbdgIRNdkm5" 2023-10-12T19:02:54.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:54.922 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_pools": [ 2023-10-12T19:02:54.922 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:54.923 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "default-placement", 2023-10-12T19:02:54.923 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:02:54.923 INFO:teuthology.orchestra.run.smithi170.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-12T19:02:54.923 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": { 2023-10-12T19:02:54.923 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD": { 2023-10-12T19:02:54.924 INFO:teuthology.orchestra.run.smithi170.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-12T19:02:54.924 INFO:teuthology.orchestra.run.smithi170.stdout: "compression_type": "random" 2023-10-12T19:02:54.924 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:54.924 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:54.924 INFO:teuthology.orchestra.run.smithi170.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-12T19:02:54.924 INFO:teuthology.orchestra.run.smithi170.stdout: "index_type": 0 2023-10-12T19:02:54.925 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:54.925 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:54.925 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:54.925 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:02:54.925 INFO:teuthology.orchestra.run.smithi170.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-12T19:02:54.925 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:54.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:02:54.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:02:54.928 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T19:02:55.007 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:02:55.007 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:02:55.054 INFO:teuthology.orchestra.run.smithi170.stderr:Sending period to new master zone cc69b071-ffc5-4662-8e68-2301691e9014 2023-10-12T19:02:56.083 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:02:56.083 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:56.084 INFO:teuthology.orchestra.run.smithi170.stdout: "epoch": 5, 2023-10-12T19:02:56.084 INFO:teuthology.orchestra.run.smithi170.stdout: "predecessor_uuid": "e1332b00-f761-4c62-a5ed-b8b60a8676fd", 2023-10-12T19:02:56.084 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": [], 2023-10-12T19:02:56.084 INFO:teuthology.orchestra.run.smithi170.stdout: "period_map": { 2023-10-12T19:02:56.085 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:02:56.085 INFO:teuthology.orchestra.run.smithi170.stdout: "zonegroups": [ 2023-10-12T19:02:56.085 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.085 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "a", 2023-10-12T19:02:56.085 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a", 2023-10-12T19:02:56.086 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "", 2023-10-12T19:02:56.086 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "true", 2023-10-12T19:02:56.086 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:02:56.086 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T19:02:56.086 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.087 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T19:02:56.087 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T19:02:56.087 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:02:56.087 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [ 2023-10-12T19:02:56.087 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.088 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T19:02:56.088 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a2", 2023-10-12T19:02:56.088 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:02:56.088 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8001" 2023-10-12T19:02:56.088 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.089 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:02:56.089 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:02:56.089 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:02:56.089 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:02:56.089 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:02:56.090 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "false", 2023-10-12T19:02:56.090 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:02:56.090 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:02:56.090 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:56.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.091 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:02:56.091 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a1", 2023-10-12T19:02:56.091 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:02:56.091 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T19:02:56.092 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.092 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:02:56.092 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:02:56.092 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:02:56.092 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:02:56.093 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:02:56.093 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:02:56.093 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:02:56.094 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:02:56.094 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:56.094 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.094 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [ 2023-10-12T19:02:56.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.095 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default-placement", 2023-10-12T19:02:56.095 INFO:teuthology.orchestra.run.smithi170.stdout: "tags": [], 2023-10-12T19:02:56.095 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": [ 2023-10-12T19:02:56.096 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD" 2023-10-12T19:02:56.096 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:56.096 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:56.096 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.096 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "default-placement", 2023-10-12T19:02:56.097 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:02:56.097 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T19:02:56.097 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T19:02:56.097 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:56.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:56.098 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.098 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "b", 2023-10-12T19:02:56.098 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "b", 2023-10-12T19:02:56.098 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "", 2023-10-12T19:02:56.098 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "false", 2023-10-12T19:02:56.099 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:02:56.099 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T19:02:56.099 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.099 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T19:02:56.099 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T19:02:56.100 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T19:02:56.100 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [ 2023-10-12T19:02:56.100 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.100 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T19:02:56.100 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "b2", 2023-10-12T19:02:56.101 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:02:56.101 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi187.front.sepia.ceph.com:8001" 2023-10-12T19:02:56.101 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.101 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:02:56.101 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:02:56.102 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:02:56.102 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:02:56.102 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:02:56.102 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:02:56.102 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:02:56.103 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:02:56.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:56.103 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.103 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T19:02:56.103 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "b1", 2023-10-12T19:02:56.104 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:02:56.104 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T19:02:56.104 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.104 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:02:56.104 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:02:56.105 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:02:56.105 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:02:56.106 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:02:56.106 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:02:56.106 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:02:56.107 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:02:56.107 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:56.107 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.107 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [ 2023-10-12T19:02:56.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.108 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default-placement", 2023-10-12T19:02:56.108 INFO:teuthology.orchestra.run.smithi170.stdout: "tags": [], 2023-10-12T19:02:56.108 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": [ 2023-10-12T19:02:56.108 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD" 2023-10-12T19:02:56.108 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:56.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:56.109 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.109 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "default-placement", 2023-10-12T19:02:56.109 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:02:56.109 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T19:02:56.110 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T19:02:56.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:56.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:56.110 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:02:56.111 INFO:teuthology.orchestra.run.smithi170.stdout: "short_zone_ids": [ 2023-10-12T19:02:56.111 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.111 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T19:02:56.111 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 3365504037 2023-10-12T19:02:56.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:56.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.112 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T19:02:56.112 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 259669302 2023-10-12T19:02:56.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:56.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.113 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T19:02:56.113 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 20676319 2023-10-12T19:02:56.113 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:56.113 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:02:56.113 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:02:56.114 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 2601847591 2023-10-12T19:02:56.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:56.114 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:02:56.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:56.115 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zonegroup": "a", 2023-10-12T19:02:56.115 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:02:56.115 INFO:teuthology.orchestra.run.smithi170.stdout: "period_config": { 2023-10-12T19:02:56.115 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_quota": { 2023-10-12T19:02:56.115 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T19:02:56.116 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T19:02:56.116 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T19:02:56.116 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T19:02:56.116 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T19:02:56.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:56.117 INFO:teuthology.orchestra.run.smithi170.stdout: "user_quota": { 2023-10-12T19:02:56.117 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T19:02:56.117 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T19:02:56.117 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T19:02:56.118 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T19:02:56.118 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T19:02:56.118 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:02:56.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:02:56.119 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:02:56.119 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_name": "test-realm", 2023-10-12T19:02:56.119 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:02:56.119 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:02:56.120 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-10-12T19:03:56.121 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-35 2023-10-12T19:03:56.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:03:56.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:03:56.156 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:03:56.203 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:03:56.203 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:03:56.242 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T19:03:56.255 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T19:03:56.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:03:56.255 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697137436.143028_223.1", 2023-10-12T19:03:56.256 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T19:03:56.143028Z" 2023-10-12T19:03:56.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:03:56.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:03:56.256 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:03:56.257 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T19:03:56.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:03:56.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:03:56.257 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:03:56.257 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:03:56.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:03:56.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:03:56.258 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:03:56.258 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:03:56.258 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:03:56.259 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T19:03:56.260 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T19:03:56.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:03:56.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:03:56.261 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:03:56.303 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:03:56.303 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:03:56.353 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:03:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:03:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:03:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:03:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:03:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:03:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:03:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:03:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:03:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:03:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:03:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:03:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:03:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:03:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:03:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:03:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:03:56.357 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:03:56.357 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:03:56.357 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:03:56.357 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:03:56.357 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:03:56.358 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:03:56.358 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:03:56.358 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:03:56.358 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:03:56.359 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:03:56.359 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:03:56.359 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:03:56.359 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:03:56.360 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:03:56.360 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:03:56.360 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:03:56.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:03:56.360 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:03:56.361 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:03:56.361 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:03:56.361 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:03:56.361 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:03:56.361 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:03:56.362 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:03:56.362 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:03:56.362 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:03:56.362 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:03:56.362 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:03:56.363 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:03:56.363 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:03:56.363 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:03:56.363 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:03:56.363 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:03:56.364 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:03:56.364 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:03:56.364 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:03:56.364 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:03:56.364 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:03:56.364 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:03:56.365 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:03:56.365 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:03:56.365 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:03:56.365 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:03:56.366 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:03:56.366 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:03:56.366 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:03:56.366 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:03:56.368 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:01.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:01.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:01.370 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:01.418 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:01.419 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:01.474 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:01.475 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:01.475 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:01.475 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:01.475 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:01.476 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:01.476 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:01.476 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:01.476 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:01.476 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:01.476 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:01.477 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:01.477 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:01.477 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:01.477 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:01.477 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:01.478 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:01.478 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:01.478 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:01.478 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:01.478 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:01.479 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:01.479 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:01.479 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:01.479 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:01.479 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:01.480 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:01.480 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:01.480 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:01.480 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:01.480 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:01.481 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:01.481 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:01.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:01.481 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:01.481 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:01.481 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:01.482 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:01.482 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:01.482 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:01.482 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:01.483 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:01.483 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:01.483 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:01.483 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:01.483 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:01.483 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:01.484 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:01.484 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:01.484 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:01.484 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:01.484 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:01.485 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:01.485 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:01.485 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:01.485 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:01.485 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:01.486 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:01.486 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:01.486 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:01.486 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:01.486 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:01.487 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:01.487 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:01.488 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:06.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:06.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:06.490 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:06.538 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:06.538 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:06.595 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:06.595 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:06.596 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:06.596 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:06.596 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:06.596 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:06.596 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:06.597 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:06.597 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:06.597 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:06.597 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:06.598 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:06.598 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:06.598 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:06.598 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:06.598 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:06.599 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:06.599 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:06.599 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:06.599 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:06.599 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:06.600 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:06.600 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:06.600 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:06.600 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:06.600 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:06.601 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:06.601 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:06.601 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:06.601 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:06.601 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:06.602 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:06.602 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:06.602 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:06.602 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:06.602 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:06.603 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:06.603 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:06.603 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:06.603 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:06.603 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:06.603 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:06.604 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:06.604 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:06.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:06.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:06.604 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:06.605 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:06.605 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:06.605 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:06.605 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:06.605 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:06.606 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:06.606 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:06.606 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:06.606 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:06.606 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:06.607 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:06.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:06.607 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:06.607 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:06.607 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:06.608 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:06.608 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:06.609 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:11.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:11.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:11.611 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:11.659 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:11.659 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:11.715 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:11.715 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:11.716 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:11.716 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:11.716 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:11.716 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:11.716 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:11.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:11.717 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:11.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:11.717 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:11.717 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:11.718 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:11.718 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:11.718 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:11.718 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:11.718 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:11.719 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:11.719 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:11.719 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:11.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:11.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:11.720 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:11.720 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:11.720 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:11.720 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:11.720 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:11.721 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:11.721 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:11.721 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:11.721 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:11.721 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:11.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:11.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:11.722 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:11.722 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:11.722 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:11.723 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:11.723 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:11.723 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:11.723 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:11.723 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:11.724 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:11.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:11.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:11.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:11.724 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:11.725 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:11.725 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:11.725 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:11.725 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:11.725 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:11.726 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:11.726 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:11.726 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:11.726 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:11.726 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:11.727 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:11.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:11.727 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:11.727 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:11.727 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:11.728 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:11.728 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:11.729 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:16.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:16.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:16.732 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:16.778 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:16.778 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:16.831 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:16.832 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:16.832 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:16.832 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:16.832 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:16.833 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:16.833 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:16.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:16.833 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:16.833 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:16.833 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:16.834 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:16.834 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:16.834 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:16.834 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:16.835 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:16.835 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:16.835 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:16.835 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:16.835 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:16.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:16.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:16.836 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:16.836 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:16.836 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:16.836 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:16.837 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:16.837 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:16.837 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:16.837 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:16.837 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:16.838 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:16.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:16.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:16.838 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:16.838 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:16.839 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:16.839 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:16.839 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:16.839 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:16.839 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:16.840 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:16.840 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:16.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:16.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:16.840 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:16.841 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:16.841 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:16.841 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:16.841 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:16.841 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:16.842 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:16.842 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:16.842 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:16.842 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:16.842 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:16.842 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:16.843 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:16.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:16.843 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:16.843 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:16.843 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:16.844 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:16.844 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:16.845 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:21.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:21.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:21.848 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:21.894 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:21.894 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:21.946 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:21.947 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:21.947 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:21.947 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:21.947 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:21.947 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:21.948 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:21.948 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:21.948 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:21.948 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:21.949 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:21.949 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:21.949 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:21.949 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:21.950 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:21.950 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:21.950 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:21.951 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:21.951 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:21.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:21.952 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:21.952 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:21.953 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:21.953 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:21.953 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:21.954 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:21.954 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:21.955 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:21.955 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:21.955 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:21.956 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:21.956 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:21.956 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:21.957 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:21.957 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:21.957 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:21.958 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:21.958 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:21.959 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:21.959 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:21.959 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:21.959 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:21.960 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:21.960 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:21.960 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:21.960 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:21.960 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:21.961 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:21.961 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:21.961 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:21.961 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:21.961 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:21.962 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:21.962 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:21.962 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:21.962 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:21.962 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:21.963 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:21.963 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:21.963 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:21.963 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:21.963 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:21.964 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:21.964 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:21.965 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:26.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:26.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:26.968 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:27.017 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:27.017 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:27.073 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:27.073 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:27.073 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:27.074 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:27.074 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:27.074 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:27.074 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:27.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:27.075 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:27.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:27.075 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:27.075 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:27.076 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:27.076 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:27.076 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:27.076 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:27.076 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:27.077 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:27.077 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:27.077 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:27.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:27.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:27.078 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:27.078 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:27.078 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:27.078 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:27.078 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:27.079 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:27.079 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:27.079 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:27.079 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:27.079 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:27.080 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:27.080 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:27.080 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:27.080 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:27.080 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:27.081 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:27.081 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:27.081 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:27.081 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:27.081 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:27.082 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:27.082 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:27.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:27.082 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:27.083 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:27.083 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:27.083 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:27.083 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:27.083 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:27.084 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:27.084 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:27.084 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:27.084 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:27.084 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:27.085 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:27.085 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:27.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:27.085 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:27.086 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:27.086 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:27.086 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:27.086 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:27.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:32.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:32.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:32.090 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:32.138 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:32.139 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:32.196 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:32.198 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:32.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:32.198 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:32.198 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:32.209 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:32.210 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:37.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:37.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:37.213 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:37.260 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:37.261 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:37.315 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:37.315 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:37.315 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:37.315 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:37.316 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:37.316 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:37.316 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:37.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:37.316 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:37.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:37.317 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:37.317 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:37.317 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:37.318 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:37.318 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:37.318 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:37.318 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:37.318 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:37.319 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:37.319 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:37.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:37.319 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:37.319 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:37.319 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:37.320 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:37.320 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:37.320 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:37.320 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:37.320 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:37.321 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:37.321 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:37.321 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:37.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:37.321 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:37.321 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:37.322 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:37.322 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:37.322 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:37.322 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:37.322 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:37.323 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:37.323 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:37.323 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:37.323 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:37.323 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:37.324 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:37.324 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:37.324 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:37.324 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:37.324 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:37.325 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:37.325 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:37.325 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:37.325 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:37.325 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:37.325 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:37.326 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:37.326 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:37.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:37.326 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:37.326 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:37.327 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:37.327 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:37.327 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:37.328 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:42.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:42.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:42.331 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:42.377 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:42.378 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:42.432 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:42.432 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:42.442 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:42.442 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:42.442 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:42.442 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:42.444 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:42.444 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:42.444 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:42.444 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:42.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:42.445 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:42.445 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:42.445 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:42.446 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:42.446 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:42.447 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:47.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:47.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:47.449 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:47.498 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:47.498 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:47.554 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:47.554 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:47.555 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:47.555 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:47.555 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:47.555 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:47.555 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:47.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:47.556 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:47.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:47.556 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:47.556 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:47.557 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:47.557 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:47.557 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:47.557 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:47.557 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:47.558 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:47.558 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:47.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:47.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:47.559 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:47.559 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:47.559 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:47.559 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:47.559 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:47.560 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:47.560 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:47.560 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:47.560 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:47.560 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:47.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:47.561 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:47.561 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:47.561 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:47.561 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:47.561 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:47.562 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:47.562 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:47.562 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:47.562 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:47.562 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:47.563 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:47.563 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:47.563 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:47.563 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:47.563 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:47.563 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:47.564 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:47.564 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:47.564 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:47.564 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:47.565 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:47.565 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:47.565 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:47.565 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:47.565 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:47.566 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:47.566 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:47.566 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:47.566 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:47.567 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:47.567 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:47.567 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:47.570 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:52.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:52.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:52.572 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:52.617 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:52.618 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:52.671 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.392700_120.1", 2023-10-12T19:04:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.392700Z", 2023-10-12T19:04:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:52.684 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:52.685 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.392700_120.1 2023-10-12T19:04:57.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:57.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:04:57.688 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:04:57.729 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:04:57.730 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:04:57.779 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:04:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:04:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:04:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:04:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:04:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:04:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:04:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:04:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:04:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:04:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:04:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:04:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:04:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:04:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:04:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:04:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:04:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:04:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:04:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:04:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:04:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:04:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:04:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:04:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:04:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:04:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:04:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:04:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:04:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:04:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:04:57.791 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:04:57.793 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.393673_121.1 2023-10-12T19:05:02.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:02.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:02.796 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:02.837 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:02.837 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:02.888 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:02.888 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:02.888 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:02.889 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:02.889 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:02.889 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:02.889 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:02.889 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:02.890 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:02.890 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:02.890 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:02.890 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:02.890 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:02.891 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:05:02.891 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:02.891 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:02.891 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:02.891 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:05:02.892 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:02.892 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:02.892 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:02.892 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:02.892 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:02.892 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:02.893 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:02.893 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:02.893 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:02.893 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:02.893 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:02.894 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:02.894 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:02.894 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:02.894 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:02.894 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:02.895 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:02.895 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:02.895 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:02.895 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:02.895 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:02.896 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:02.896 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:02.896 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:02.896 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:02.896 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:02.897 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:02.897 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:02.897 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:02.897 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:02.897 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:02.898 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:02.898 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:02.898 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:02.898 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:02.898 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:02.898 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:02.899 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:02.899 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:02.899 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:02.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:02.899 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:02.900 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:02.900 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:02.900 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:02.900 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:02.901 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:05:07.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:07.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:07.904 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:07.994 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:07.995 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:08.104 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:08.104 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:08.105 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:08.105 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:08.105 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:08.105 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:08.105 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:08.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:08.106 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:08.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:08.106 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:08.106 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:08.107 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:08.107 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:05:08.107 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:08.107 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:08.107 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:08.108 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:05:08.108 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:08.108 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:08.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:08.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:08.109 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:08.109 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:08.109 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:08.109 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:08.109 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:08.110 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:08.110 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:08.110 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:08.110 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:08.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:08.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:08.111 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:08.111 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:08.111 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:08.111 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:08.112 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:08.112 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:08.112 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:08.112 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:08.112 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:08.113 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:08.113 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:08.113 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:08.113 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:08.113 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:08.114 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:08.114 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:08.114 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:08.114 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:08.114 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:08.115 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:08.115 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:08.115 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:08.115 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:08.115 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:08.116 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:08.116 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:08.116 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:08.116 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:08.116 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:08.117 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:08.117 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:08.118 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.393673_121.1 2023-10-12T19:05:13.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:13.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:13.120 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:13.214 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:13.214 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:13.301 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:13.301 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:13.302 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:13.302 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:13.302 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:13.302 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:13.302 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:13.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:13.303 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:13.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:13.303 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:13.303 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:13.304 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:13.304 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.611004_153.1", 2023-10-12T19:05:13.304 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:13.304 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:13.304 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:13.305 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.611004Z", 2023-10-12T19:05:13.305 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:13.305 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:13.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:13.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:13.306 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:13.306 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:13.306 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:13.306 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:13.307 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:13.307 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:13.307 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:13.307 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:13.307 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:13.307 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:13.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:13.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:13.308 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:13.308 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:13.308 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:13.309 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:13.309 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:13.309 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:13.309 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:13.309 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:13.310 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:13.310 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:13.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:13.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:13.310 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:13.311 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:13.311 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:13.311 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:13.311 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:13.311 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:13.312 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:13.312 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:13.312 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:13.312 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:13.312 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:13.313 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:13.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:13.313 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:13.313 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:13.313 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:13.313 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:13.314 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:13.315 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.611004_153.1 2023-10-12T19:05:18.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:18.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:18.317 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:18.359 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:18.359 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:18.409 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:18.409 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:18.409 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:18.410 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:18.410 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:18.410 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:18.410 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:18.411 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:05:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:18.412 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:05:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:18.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:18.414 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:18.415 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:18.417 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:18.418 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:18.419 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:18.420 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:18.421 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:18.421 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:18.421 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:18.421 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:18.421 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:18.422 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:18.423 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.467824_131.1 2023-10-12T19:05:23.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:23.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:23.426 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:23.475 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:23.476 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:23.525 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:23.525 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:23.525 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:23.525 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:23.526 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:23.526 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:23.526 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:23.526 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:23.527 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:23.527 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:23.527 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:23.527 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:23.527 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:23.528 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:05:23.528 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:23.528 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:23.528 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:23.528 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:05:23.529 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:23.529 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:23.529 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:23.529 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:23.529 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:23.530 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:23.530 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:23.530 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:23.530 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:23.530 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:23.531 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:23.531 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:23.531 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:23.531 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:23.531 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:23.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:23.532 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:23.532 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:23.532 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:23.532 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:23.532 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:23.533 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:23.533 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:23.533 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:23.533 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:23.533 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:23.534 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:23.534 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:23.534 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:23.534 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:23.534 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:23.534 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:23.535 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:23.535 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:23.535 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:23.535 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:23.536 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:23.536 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:23.536 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:23.536 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:23.536 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:23.536 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:23.537 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:23.537 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:23.537 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:23.537 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:23.538 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:05:28.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:28.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:28.541 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:28.632 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:28.633 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:28.814 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:28.815 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:28.815 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:28.815 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:28.815 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:28.816 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:28.816 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:28.816 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:28.816 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:28.817 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:28.817 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:28.817 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:28.817 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:28.817 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:05:28.818 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:28.818 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:28.818 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:28.818 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:05:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:28.820 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:28.820 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:28.820 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:28.821 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:28.821 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:28.822 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:28.822 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:28.822 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:28.823 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:28.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:28.823 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:28.823 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:28.827 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:28.827 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:28.828 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:28.828 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:28.828 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:28.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:28.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:28.829 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:28.830 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:28.830 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:28.830 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:28.831 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:28.831 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:28.831 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:28.833 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.396100_122.1 2023-10-12T19:05:33.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:33.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:33.836 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:33.901 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:33.902 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:33.951 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:33.951 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:33.951 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:33.953 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:33.953 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:33.953 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:33.953 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:33.954 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:33.954 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:05:33.954 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:33.954 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:33.955 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:33.955 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:05:33.956 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:33.956 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:33.956 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:33.956 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:33.956 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:33.957 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:33.957 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:33.957 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:33.957 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:33.957 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:33.958 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:33.958 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:33.958 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:33.958 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:33.958 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:33.959 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:33.959 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:33.959 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:33.959 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:33.959 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:33.960 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:33.960 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:33.960 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:33.960 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:33.960 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:33.961 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:33.961 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:33.961 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:33.961 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:33.961 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:33.962 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:33.962 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:33.962 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:33.962 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:33.962 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:33.963 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:33.963 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:33.963 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:33.963 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:33.964 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:33.964 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:33.964 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:33.964 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:33.965 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:33.965 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:33.965 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:33.967 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.410419_126.1 2023-10-12T19:05:38.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:38.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:38.970 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:39.054 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:39.054 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:39.134 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:39.134 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:39.135 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:39.135 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:39.135 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:39.135 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:39.135 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:39.136 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:39.136 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:39.136 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:39.137 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:39.137 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:39.137 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:39.137 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:05:39.137 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:39.138 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:39.138 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:39.138 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:05:39.138 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:39.138 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:39.139 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:39.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:39.139 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:39.139 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:39.140 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:39.140 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:39.140 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:39.140 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:39.140 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:39.141 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:39.141 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:39.141 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:39.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:39.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:39.142 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:39.142 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:39.142 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:39.142 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:39.143 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:39.143 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:39.143 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:39.143 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:39.143 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:39.144 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:39.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:39.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:39.144 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:39.144 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:39.145 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:39.145 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:39.145 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:39.145 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:39.146 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:39.146 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:39.146 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:39.146 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:39.147 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:39.147 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:39.147 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:39.147 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:39.148 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:39.148 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:39.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:39.148 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:39.149 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:05:44.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:44.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:44.151 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:44.211 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:44.212 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:44.267 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:44.267 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:44.267 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:44.268 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:44.268 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:44.268 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:44.268 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:44.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:44.269 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:44.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:44.269 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:44.269 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:44.270 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:44.270 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:05:44.270 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:44.270 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:05:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:44.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:44.272 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:44.272 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:44.278 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:44.278 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:44.278 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:44.278 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:44.281 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:44.282 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.393673_121.1 2023-10-12T19:05:49.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:49.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:49.284 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:49.332 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:49.332 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:49.382 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:49.382 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:49.382 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:49.382 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:49.383 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:49.383 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:49.383 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:49.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:49.383 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:49.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:49.384 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:49.384 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:49.384 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:49.384 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.571704_145.1", 2023-10-12T19:05:49.385 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:49.385 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:49.385 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:49.385 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.571704Z", 2023-10-12T19:05:49.385 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:49.386 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:49.386 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:49.386 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:49.386 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:49.386 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:49.386 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:49.387 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:49.387 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:49.387 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:49.387 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:49.387 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:49.388 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:49.388 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:49.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:49.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:49.388 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:49.389 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:49.389 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:49.389 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:49.389 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:49.389 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:49.390 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:49.390 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:49.390 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:49.390 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:49.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:49.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:49.391 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:49.391 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:49.391 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:49.391 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:49.391 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:49.394 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:49.394 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:49.395 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.571704_145.1 2023-10-12T19:05:54.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:54.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:54.397 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:54.477 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:54.477 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:54.555 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:54.555 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:54.555 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:54.556 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:54.556 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:54.556 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:54.556 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:54.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:54.557 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:54.557 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:54.557 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:54.557 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:54.557 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:54.558 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T19:05:54.558 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:54.558 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:54.558 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:54.558 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T19:05:54.559 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:54.559 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:54.559 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:54.559 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:54.559 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:54.560 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:54.560 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:54.560 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:54.560 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:54.560 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:54.561 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:54.561 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:54.561 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:54.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:54.561 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:54.562 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:54.562 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:54.562 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:54.562 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:54.562 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:54.563 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:54.563 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:54.563 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:54.563 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:54.563 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:54.564 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:54.564 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:54.564 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:54.564 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:54.564 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:54.565 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:54.565 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:54.565 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:54.565 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:54.565 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:54.566 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:54.566 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:54.566 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:54.566 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:54.566 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:54.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:54.567 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:54.567 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:54.567 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:54.567 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:54.568 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:54.569 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.473047_134.1 2023-10-12T19:05:59.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:59.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:05:59.571 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:05:59.644 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:05:59.645 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:05:59.728 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:05:59.728 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:05:59.728 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:05:59.729 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:05:59.729 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:05:59.729 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:05:59.729 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:59.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:59.730 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:05:59.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:59.730 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:05:59.730 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:59.731 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:59.731 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:05:59.731 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:59.731 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:05:59.731 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:59.732 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:05:59.732 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:59.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:59.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:59.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:59.733 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:05:59.733 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:59.733 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:59.733 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:05:59.733 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:59.734 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:59.734 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:59.734 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:05:59.734 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:59.734 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:59.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:59.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:59.735 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:05:59.735 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:59.735 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:59.735 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:59.736 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:59.736 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:59.736 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:59.736 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:59.736 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:59.737 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:59.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:59.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:05:59.737 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:05:59.737 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:05:59.738 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:05:59.738 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:05:59.738 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:05:59.738 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:05:59.738 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:05:59.739 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:05:59.739 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:05:59.739 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:59.739 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:59.739 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:05:59.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:05:59.740 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:05:59.740 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:05:59.740 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:05:59.740 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:05:59.741 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:05:59.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:06:04.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:04.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:04.744 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:04.835 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:04.835 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:05.053 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:05.054 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:05.054 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:05.054 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:05.055 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:05.055 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:05.055 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:05.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:05.055 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:05.056 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:05.056 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:05.056 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:05.056 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:05.056 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:06:05.057 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:05.057 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:05.057 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:05.057 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:06:05.057 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:05.058 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:05.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:05.058 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:05.058 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:05.058 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:05.059 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:05.059 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:05.059 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:05.059 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:05.059 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:05.060 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:05.060 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:05.060 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:05.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:05.060 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:05.061 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:05.061 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:05.061 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:05.061 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:05.061 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:05.061 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:05.062 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:05.062 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:05.062 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:05.062 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:05.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:05.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:05.063 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:05.063 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:05.063 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:05.063 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:05.064 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:05.065 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:05.065 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:05.065 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:05.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:05.065 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:05.067 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:05.067 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:05.067 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:05.068 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:05.069 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.410419_126.1 2023-10-12T19:06:10.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:10.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:10.072 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:10.159 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:10.160 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:10.310 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:10.310 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:10.311 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:10.311 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:10.311 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:10.311 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:10.311 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:10.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:10.312 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:10.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:10.312 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:10.312 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:10.313 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:10.313 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:06:10.313 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:10.313 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:10.313 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:10.314 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:06:10.314 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:10.314 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:10.314 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:10.314 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:10.315 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:10.315 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:10.315 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:10.315 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:10.316 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:10.316 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:10.316 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:10.316 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:10.316 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:10.316 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:10.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:10.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:10.317 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:10.317 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:10.317 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:10.318 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:10.318 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:10.318 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:10.318 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:10.318 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:10.319 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:10.319 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:10.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:10.319 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:10.319 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:10.320 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:10.320 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:10.320 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:10.320 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:10.320 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:10.321 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:10.321 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:10.321 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:10.321 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:10.321 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:10.322 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:10.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:10.322 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:10.322 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:10.322 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:10.323 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:10.323 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:10.324 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:06:15.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:15.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:15.326 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:15.412 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:15.413 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:15.495 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:15.495 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:15.496 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:15.496 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:15.496 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:15.496 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:15.496 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:15.497 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:15.497 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:15.497 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:15.497 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:15.497 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:15.498 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:15.498 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:06:15.498 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:15.498 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:15.498 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:15.499 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:06:15.499 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:15.499 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:15.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:15.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:15.500 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:15.500 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:15.500 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:15.500 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:15.500 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:15.500 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:15.501 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:15.501 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:15.501 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:15.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:15.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:15.501 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:15.502 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:15.502 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:15.502 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:15.502 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:15.502 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:15.503 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:15.503 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:15.503 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:15.503 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:15.503 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:15.503 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:15.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:15.504 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:15.504 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:15.504 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:15.504 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:15.504 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:15.507 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:15.507 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:15.507 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:15.508 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.435183_128.1 2023-10-12T19:06:20.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:20.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:20.510 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:20.563 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:20.563 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:20.616 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:20.616 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:20.616 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:20.616 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:20.617 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:20.617 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:20.617 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:20.617 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:20.617 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:20.618 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:20.618 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:20.618 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:20.618 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:20.619 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:06:20.619 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:20.619 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:20.619 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:20.619 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:06:20.620 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:20.620 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:20.620 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:20.620 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:20.620 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:20.621 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:20.621 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:20.621 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:20.621 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:20.621 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:20.622 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:20.622 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:20.622 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:20.622 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:20.623 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:20.623 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:20.623 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:20.623 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:20.623 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:20.623 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:20.624 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:20.624 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:20.624 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:20.624 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:20.624 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:20.624 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:20.625 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:20.625 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:20.625 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:20.625 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:20.625 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:20.626 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:20.626 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:20.626 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:20.627 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:20.627 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:20.627 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:20.628 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:20.628 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:20.628 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:20.628 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:20.629 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:20.629 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:20.629 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:20.629 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:20.629 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:20.631 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.396100_122.1 2023-10-12T19:06:25.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:25.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:25.633 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:25.675 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:25.675 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:25.725 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:25.725 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:25.726 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:25.726 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:25.726 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:25.726 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:25.726 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:25.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:25.727 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:25.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:25.727 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:25.727 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:25.728 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:25.728 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:06:25.728 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:25.728 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:25.728 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:25.729 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:06:25.729 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:25.729 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:25.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:25.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:25.730 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:25.730 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:25.730 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:25.730 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:25.730 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:25.731 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:25.731 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:25.731 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:25.731 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:25.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:25.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:25.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:25.732 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:25.732 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:25.732 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:25.732 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:25.733 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:25.733 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:25.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:25.733 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:25.733 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:25.734 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:25.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:25.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:25.734 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:25.734 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:25.735 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:25.735 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:25.735 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:25.735 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:25.735 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:25.736 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:25.736 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:25.736 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:25.736 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:25.736 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:25.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:25.737 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:25.737 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:25.737 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:25.737 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:25.738 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:25.739 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:06:30.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:30.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:30.741 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:30.807 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:30.807 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:30.868 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:30.868 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:30.869 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:30.869 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:30.869 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:30.869 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:30.869 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:30.870 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:30.870 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:30.870 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:30.871 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:30.871 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:30.871 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:30.871 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:06:30.872 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:30.872 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:30.872 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:30.872 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:06:30.872 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:30.873 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:30.873 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:30.873 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:30.873 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:30.873 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:30.874 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:30.874 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:30.874 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:30.874 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:30.874 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:30.875 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:30.875 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:30.875 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:30.875 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:30.875 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:30.875 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:30.876 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:30.876 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:30.876 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:30.876 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:30.876 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:30.877 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:30.877 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:30.877 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:30.877 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:30.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:30.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:30.878 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:30.878 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:30.878 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:30.878 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:30.879 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:30.879 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:30.879 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:30.879 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:30.880 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:30.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:30.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:30.880 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:30.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:30.881 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:30.881 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:30.881 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:30.881 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:30.882 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:30.883 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:06:35.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:35.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:35.886 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:35.928 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:35.928 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:35.979 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:35.979 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:35.979 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:35.980 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:35.980 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:35.980 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:35.980 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:35.980 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:35.981 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:35.981 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:35.981 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:35.981 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:35.982 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:35.982 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:06:35.982 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:35.982 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:35.982 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:35.983 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:06:35.983 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:35.983 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:35.983 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:35.983 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:35.984 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:35.984 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:35.984 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:35.984 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:35.984 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:35.984 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:35.985 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:35.985 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:35.985 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:35.985 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:35.985 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:35.986 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:35.986 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:35.986 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:35.986 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:35.986 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:35.987 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:35.987 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:35.987 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:35.987 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:35.987 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:35.988 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:35.988 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:35.988 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:35.988 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:35.988 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:35.989 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:35.989 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:35.989 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:35.989 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:35.990 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:35.990 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:35.990 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:35.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:35.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:35.991 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:35.991 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:35.991 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:35.991 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:35.991 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:35.992 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:35.992 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:35.993 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.393673_121.1 2023-10-12T19:06:40.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:40.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:40.995 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:41.073 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:41.073 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:41.134 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:41.134 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:41.134 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:06:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:41.138 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:06:41.138 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:41.138 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:41.139 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:41.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:41.139 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:41.139 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:41.142 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:41.142 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:41.142 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:41.142 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:41.142 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:41.143 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:41.143 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:41.143 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:41.143 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:41.148 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:41.149 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:06:46.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:46.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:46.152 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:46.235 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:46.235 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:46.298 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:46.298 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:46.298 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:46.299 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:46.299 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:46.299 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:46.300 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:46.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:46.300 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:46.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:46.301 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:46.301 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:46.301 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:46.302 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:06:46.302 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:46.302 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:46.302 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:46.302 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:06:46.303 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:46.303 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:46.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:46.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:46.303 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:46.304 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:46.304 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:46.304 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:46.304 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:46.304 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:46.305 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:46.305 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:46.305 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:46.305 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:46.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:46.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:46.306 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:46.306 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:46.306 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:46.306 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:46.307 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:46.307 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:46.308 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:46.308 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:46.308 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:46.309 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:46.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:46.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:46.309 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:46.309 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:46.310 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:46.310 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:46.310 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:46.310 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:46.310 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:46.311 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:46.311 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:46.311 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:46.311 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:46.311 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:46.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:46.312 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:46.312 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:46.312 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:46.312 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:46.313 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:46.314 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:06:51.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:51.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:51.317 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:51.359 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:51.360 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:51.411 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:51.411 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:51.411 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:51.412 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:51.412 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:51.412 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:51.412 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:51.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:51.413 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:51.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:51.413 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:51.413 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:51.413 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:51.413 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:06:51.414 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:51.414 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:51.414 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:51.414 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:06:51.415 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:51.415 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:51.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:51.415 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:51.415 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:51.416 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:51.416 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:51.416 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:51.416 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:51.416 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:51.417 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:51.417 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:51.417 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:51.417 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:51.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:51.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:51.418 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:51.418 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:51.418 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:51.418 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:51.419 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:51.419 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:51.419 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:51.419 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:51.419 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:51.420 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:51.420 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:51.420 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:51.420 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:51.420 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:51.421 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:51.421 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:51.421 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:51.421 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:51.421 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:51.422 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:51.422 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:51.422 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:51.422 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:51.422 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:51.423 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:51.423 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:51.423 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:51.423 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:51.423 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:51.424 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:51.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.396100_122.1 2023-10-12T19:06:56.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:56.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:06:56.427 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:06:56.514 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:06:56.514 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:06:56.638 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:06:56.638 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:06:56.639 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:06:56.639 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:06:56.639 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:06:56.639 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:06:56.640 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:56.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:56.640 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:06:56.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:56.640 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:06:56.640 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:56.641 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:56.641 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:06:56.641 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:56.641 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:06:56.641 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:56.642 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:06:56.642 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:56.642 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:56.642 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:56.643 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:56.643 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:06:56.643 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:56.643 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:56.643 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:06:56.643 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:56.644 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:56.644 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:56.644 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:06:56.644 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:56.644 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:56.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:56.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:56.645 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:06:56.645 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:56.645 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:56.646 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:56.646 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:56.646 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:56.646 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:56.646 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:56.647 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:56.647 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:56.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:56.647 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:06:56.647 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:06:56.648 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:06:56.648 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:06:56.648 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:06:56.648 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:06:56.648 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:06:56.649 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:06:56.649 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:06:56.649 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:06:56.649 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:56.649 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:56.650 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:06:56.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:06:56.650 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:06:56.650 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:06:56.650 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:06:56.651 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:06:56.651 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:06:56.652 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.432502_127.1 2023-10-12T19:07:01.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:01.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:01.654 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:01.736 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:01.737 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:01.981 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:01.981 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:01.981 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:01.982 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:01.982 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:01.982 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:01.982 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:01.982 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:01.983 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:01.983 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:01.983 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:01.983 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:01.983 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:01.984 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.486122_135.1", 2023-10-12T19:07:01.984 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:01.984 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:01.984 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:01.984 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.486122Z", 2023-10-12T19:07:01.985 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:01.985 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:01.985 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:01.985 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:01.985 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:01.986 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:01.986 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:01.986 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:01.986 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:01.986 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:01.987 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:01.987 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:01.987 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:01.987 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:01.987 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:01.987 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:01.988 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:01.988 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:01.988 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:01.988 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:01.988 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:01.989 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:01.989 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:01.989 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:01.989 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:01.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:01.990 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:01.990 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:01.990 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:01.990 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:01.991 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:01.991 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:01.991 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:01.991 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:01.991 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:01.993 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:01.994 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.486122_135.1 2023-10-12T19:07:06.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:06.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:06.997 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:07.043 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:07.097 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:07.097 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:07.098 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:07.098 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:07.098 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:07.098 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.495204_138.1", 2023-10-12T19:07:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.495204Z", 2023-10-12T19:07:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:07.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:07.102 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:07.102 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:07.102 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:07.106 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:07.106 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:07.106 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:07.106 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:07.109 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:07.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:07.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:07.109 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:07.111 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:07.112 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.495204_138.1 2023-10-12T19:07:12.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:12.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:12.114 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:12.192 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:12.192 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:12.286 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:12.286 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:12.287 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:12.287 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:12.287 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:12.287 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:12.287 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:12.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:12.288 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:12.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:12.288 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:12.288 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:12.288 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:12.289 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:07:12.289 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:12.289 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:12.289 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:12.289 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:07:12.290 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:12.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:12.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:12.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:12.291 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:12.291 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:12.291 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:12.291 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:12.291 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:12.292 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:12.292 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:12.292 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:12.292 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:12.292 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:12.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:12.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:12.293 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:12.293 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:12.293 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:12.293 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:12.294 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:12.294 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:12.294 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:12.294 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:12.294 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:12.295 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:12.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:12.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:12.295 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:12.295 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:12.296 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:12.296 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:12.296 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:12.296 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:12.296 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:12.297 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:12.297 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:12.297 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:12.297 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:12.297 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:12.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:12.298 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:12.298 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:12.298 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:12.298 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:12.299 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:12.300 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:07:17.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:17.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:17.303 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:17.345 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:17.346 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:17.416 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:17.416 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:17.416 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:17.417 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:17.417 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:17.417 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:17.417 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:17.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:17.418 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:17.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:17.418 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:17.418 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:17.418 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:17.419 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:07:17.419 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:17.419 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:17.419 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:17.419 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:07:17.420 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:17.420 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:17.420 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:17.420 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:17.420 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:17.421 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:17.421 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:17.421 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:17.421 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:17.421 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:17.422 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:17.422 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:17.422 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:17.422 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:17.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:17.423 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:17.423 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:17.423 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:17.423 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:17.424 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:17.424 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:17.424 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:17.424 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:17.424 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:17.425 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:17.425 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:17.425 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:17.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:17.425 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:17.425 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:17.426 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:17.426 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:17.426 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:17.426 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:17.427 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:17.427 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:17.427 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:17.427 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:17.427 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:17.428 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:17.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:17.428 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:17.428 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:17.428 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:17.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:17.429 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:17.430 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.393673_121.1 2023-10-12T19:07:22.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:22.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:22.433 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:22.487 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:22.488 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:22.751 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:22.751 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:22.752 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:22.753 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:07:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:22.754 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:22.755 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:07:22.755 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:22.755 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:22.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:22.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:22.756 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:22.756 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:22.756 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:22.756 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:22.757 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:22.757 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:22.757 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:22.757 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:22.757 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:22.758 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:22.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:22.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:22.758 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:22.758 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:22.759 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:22.759 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:22.759 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:22.759 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:22.759 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:22.760 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:22.760 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:22.760 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:22.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:22.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:22.761 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:22.761 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:22.761 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:22.761 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:22.761 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:22.762 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:22.762 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:22.762 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:22.762 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:22.762 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:22.763 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:22.763 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:22.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:22.763 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:22.763 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:22.764 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:22.764 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:22.764 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:22.764 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.435183_128.1 2023-10-12T19:07:27.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:27.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:27.767 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:27.841 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:27.841 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:27.941 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:27.941 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:27.942 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:27.942 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:27.942 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:27.942 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:27.943 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:27.943 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:27.943 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:27.943 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:27.943 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:27.943 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:27.944 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:27.944 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:07:27.944 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:27.945 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:27.945 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:27.945 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:07:27.945 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:27.945 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:27.946 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:27.946 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:27.946 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:27.946 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:27.946 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:27.947 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:27.947 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:27.947 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:27.947 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:27.947 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:27.947 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:27.948 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:27.948 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:27.948 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:27.948 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:27.948 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:27.949 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:27.949 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:27.949 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:27.949 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:27.949 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:27.950 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:27.954 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:27.955 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.396100_122.1 2023-10-12T19:07:32.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:32.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:32.959 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:33.028 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:33.029 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:33.104 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:33.104 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:33.104 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:33.105 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:33.105 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:33.105 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:33.105 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:33.105 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:33.106 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:33.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:33.106 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:33.106 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:33.107 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:33.107 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:07:33.107 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:33.107 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:33.107 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:33.108 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:07:33.108 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:33.108 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:33.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:33.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:33.109 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:33.109 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:33.109 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:33.109 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:33.109 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:33.110 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:33.110 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:33.110 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:33.110 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:33.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:33.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:33.111 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:33.111 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:33.111 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:33.112 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:33.112 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:33.112 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:33.112 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:33.113 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:33.113 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:33.113 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:33.113 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:33.113 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:33.114 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:33.114 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:33.114 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:33.114 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:33.115 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:33.115 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:33.115 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:33.116 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:33.116 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:33.116 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:33.116 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:33.116 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:33.116 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:33.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:33.117 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:33.117 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:33.117 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:33.117 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:33.118 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:33.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.467824_131.1 2023-10-12T19:07:38.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:38.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:38.121 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:38.203 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:38.203 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:38.310 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:38.313 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:38.314 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:38.314 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:38.314 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:38.314 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:38.314 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:38.315 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:38.315 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:38.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:38.315 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:38.315 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:38.316 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:38.316 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:07:38.316 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:38.316 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:38.317 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:38.317 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:07:38.317 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:38.317 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:38.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:38.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:38.318 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:38.318 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:38.319 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:38.319 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:38.319 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:38.319 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:38.319 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:38.320 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:38.320 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:38.320 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:38.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:38.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:38.321 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:38.321 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:38.321 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:38.321 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:38.321 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:38.322 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:38.322 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:38.322 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:38.322 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:38.322 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:38.323 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:38.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:38.323 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:38.323 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:38.323 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:38.324 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:38.324 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:38.324 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:38.324 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:38.324 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:38.325 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:38.325 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:38.325 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:38.325 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:38.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:38.326 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:38.326 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:38.326 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:38.327 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:38.327 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:38.328 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.470944_133.1 2023-10-12T19:07:43.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:43.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:43.330 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:43.426 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:43.426 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:43.594 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:43.594 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:43.594 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:43.595 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:43.595 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:43.595 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:43.595 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:43.595 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:43.596 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:43.596 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:43.596 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:43.596 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:43.596 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:43.597 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:07:43.597 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:43.597 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:43.597 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:43.597 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:07:43.598 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:43.598 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:43.598 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:43.598 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:43.598 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:43.599 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:43.599 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:43.599 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:43.599 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:43.600 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:43.600 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:43.600 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:43.600 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:43.600 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:43.601 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:43.601 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:43.601 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:43.601 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:43.601 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:43.602 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:43.602 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:43.602 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:43.602 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:43.602 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:43.603 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:43.603 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:43.603 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:43.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:43.603 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:43.603 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:43.604 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:43.604 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:43.604 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:43.604 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:43.604 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:43.605 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:43.605 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:43.605 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:43.605 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:43.605 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:43.606 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:43.606 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:43.606 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:43.606 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:43.606 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:43.607 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:43.608 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.393673_121.1 2023-10-12T19:07:48.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:48.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:48.610 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:48.683 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:48.683 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:48.754 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:48.754 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:48.754 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:48.754 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:48.755 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:48.755 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:48.755 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:48.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:48.755 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:48.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:48.756 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:48.756 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:48.756 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:48.756 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:07:48.757 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:48.757 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:48.757 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:48.757 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:07:48.757 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:48.758 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:48.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:48.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:48.758 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:48.758 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:48.759 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:48.759 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:48.759 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:48.759 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:48.759 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:48.760 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:48.760 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:48.760 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:48.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:48.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:48.761 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:48.761 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:48.761 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:48.761 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:48.761 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:48.762 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:48.762 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:48.762 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:48.762 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:48.762 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:48.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:48.763 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:48.763 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:48.763 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:48.763 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:48.764 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:48.764 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:48.764 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:48.764 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:48.764 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:48.765 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:48.765 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:48.765 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:48.765 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:48.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:48.766 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:48.766 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:48.766 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:48.766 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:48.766 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:48.767 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.467824_131.1 2023-10-12T19:07:53.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:53.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:53.770 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:53.840 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:53.840 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:53.919 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:53.920 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:53.920 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:53.920 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:53.920 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:53.921 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:53.921 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:53.921 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:53.921 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:53.921 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:53.922 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:53.922 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:53.922 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:53.922 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:07:53.922 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:53.923 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:53.923 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:53.923 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:07:53.923 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:53.923 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:53.924 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:53.924 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:53.924 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:53.924 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:53.924 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:53.925 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:53.925 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:53.925 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:53.925 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:53.925 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:53.925 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:53.926 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:53.926 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:53.926 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:53.926 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:53.927 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:53.927 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:53.927 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:53.927 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:53.927 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:53.928 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:53.928 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:53.928 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:53.928 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:53.929 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:53.929 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:53.929 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:53.929 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:53.929 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:53.929 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:53.930 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:53.930 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:53.930 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:53.930 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:53.930 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:53.931 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:53.931 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:53.931 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:53.931 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:53.931 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:53.931 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:53.932 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:53.932 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:53.932 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:53.933 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:07:58.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:58.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:07:58.936 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:07:59.025 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:07:59.025 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:07:59.093 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:07:59.093 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:07:59.094 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:07:59.094 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:07:59.094 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:07:59.094 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:07:59.094 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:59.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:59.095 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:07:59.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:59.095 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:07:59.095 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:59.096 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:59.096 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:07:59.096 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:59.096 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:07:59.096 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:59.097 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:07:59.097 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:59.097 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:59.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:59.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:59.098 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:07:59.098 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:59.098 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:59.098 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:07:59.098 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:59.099 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:59.099 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:59.099 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:07:59.099 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:59.099 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:59.100 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:59.100 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:59.100 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:07:59.100 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:59.100 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:59.101 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:59.101 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:59.101 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:59.101 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:59.101 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:59.102 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:59.102 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:59.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:59.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:07:59.102 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:07:59.102 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:07:59.103 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:07:59.103 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:07:59.103 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:07:59.103 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:07:59.103 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:07:59.104 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:07:59.104 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:07:59.104 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:59.104 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:59.104 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:07:59.105 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:07:59.105 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:07:59.105 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:07:59.105 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:07:59.105 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:07:59.105 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:07:59.106 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.396100_122.1 2023-10-12T19:08:04.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:04.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:04.109 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:04.167 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:04.168 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:04.227 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:04.227 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:04.228 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:04.228 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:04.228 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:04.228 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:04.229 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:04.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:04.229 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:04.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:04.229 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:04.230 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:04.230 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:04.230 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:08:04.230 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:04.230 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:04.231 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:04.231 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:08:04.231 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:04.231 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:04.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:04.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:04.232 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:04.232 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:04.232 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:04.232 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:04.233 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:04.233 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:04.233 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:04.233 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:04.234 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:04.234 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:04.234 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:04.234 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:04.235 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:04.235 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:04.235 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:04.235 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:04.235 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:04.235 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:04.236 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:04.236 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:04.236 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:04.236 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:04.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:04.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:04.237 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:04.237 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:04.237 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:04.237 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:04.238 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:04.238 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:04.238 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:04.238 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:04.238 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:04.239 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:04.239 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:04.239 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:04.240 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:04.240 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:04.240 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:04.240 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:04.240 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:04.241 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:04.242 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:08:09.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:09.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:09.244 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:09.323 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:09.323 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:09.390 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:09.390 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:09.391 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:09.391 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:09.391 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:09.391 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:09.392 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:09.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:09.392 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:09.392 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:09.392 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:09.393 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:09.393 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:09.393 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:08:09.393 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:09.393 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:09.394 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:09.394 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:08:09.394 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:09.394 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:09.394 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:09.395 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:09.395 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:09.395 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:09.395 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:09.395 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:09.395 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:09.396 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:09.396 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:09.396 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:09.396 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:09.397 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:09.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:09.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:09.397 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:09.397 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:09.398 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:09.398 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:09.398 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:09.398 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:09.398 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:09.399 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:09.399 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:09.399 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:09.399 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:09.399 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:09.399 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:09.400 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:09.400 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:09.400 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:09.400 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:09.400 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:09.401 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:09.401 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:09.401 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:09.401 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:09.401 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:09.402 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:09.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:09.402 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:09.402 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:09.402 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:09.403 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:09.403 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:09.404 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.470944_133.1 2023-10-12T19:08:14.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:14.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:14.406 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:14.448 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:14.449 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:14.499 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:14.500 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:14.500 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:14.500 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:14.500 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:14.501 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:14.501 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:14.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:14.501 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:14.501 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:14.502 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:14.502 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:14.502 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:14.502 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:08:14.502 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:14.503 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:14.503 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:14.503 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:08:14.503 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:14.503 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:14.504 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:14.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:14.504 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:14.504 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:14.504 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:14.504 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:14.505 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:14.505 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:14.505 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:14.505 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:14.505 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:14.506 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:14.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:14.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:14.506 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:14.506 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:14.507 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:14.507 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:14.507 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:14.507 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:14.507 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:14.508 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:14.508 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:14.508 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:14.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:14.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:14.509 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:14.509 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:14.509 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:14.509 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:14.509 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:14.510 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:14.510 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:14.510 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:14.510 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:14.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:14.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:14.511 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:14.511 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:14.511 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:14.511 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:14.511 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:14.512 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:14.512 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:14.513 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:08:19.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:19.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:19.515 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:19.558 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:19.558 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:19.608 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:19.608 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:19.608 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:19.609 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:19.609 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:19.609 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:19.609 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:19.610 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:19.610 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:19.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:19.610 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:19.610 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:19.611 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:19.611 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:08:19.611 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:19.611 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:19.611 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:19.612 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:08:19.612 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:19.612 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:19.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:19.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:19.613 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:19.613 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:19.613 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:19.613 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:19.613 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:19.614 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:19.614 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:19.614 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:19.614 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:19.614 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:19.615 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:19.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:19.615 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:19.615 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:19.615 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:19.616 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:19.616 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:19.616 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:19.616 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:19.616 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:19.617 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:19.617 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:19.617 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:19.617 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:19.617 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:19.617 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:19.618 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:19.618 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:19.618 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:19.618 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:19.619 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:19.619 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:19.619 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:19.619 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:19.619 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:19.620 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:19.620 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:19.620 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:19.620 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:19.620 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:19.621 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:19.621 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:19.622 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:08:24.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:24.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:24.624 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:24.667 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:24.668 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:24.719 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:08:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:08:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:24.733 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:24.734 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.393673_121.1 2023-10-12T19:08:29.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:29.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:29.737 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:29.779 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:29.779 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:29.829 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:29.830 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:29.830 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:29.830 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:08:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:08:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:29.840 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:29.840 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:29.840 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:29.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:29.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:29.841 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:29.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:29.841 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:29.841 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:29.841 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:29.841 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:29.842 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:29.843 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.432502_127.1 2023-10-12T19:08:34.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:34.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:34.846 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:34.889 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:34.889 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:34.941 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:34.941 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:34.942 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:34.942 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:34.942 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:34.942 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:34.943 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:34.943 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:34.943 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:34.943 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:34.943 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:34.944 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:34.944 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:34.944 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T19:08:34.944 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:34.944 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:34.945 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:34.945 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T19:08:34.945 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:34.945 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:34.945 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:34.946 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:34.946 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:34.946 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:34.946 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:34.946 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:34.947 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:34.947 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:34.947 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:34.947 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:34.947 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:34.948 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:34.948 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:34.948 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:34.948 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:34.948 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:34.949 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:34.949 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:34.949 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:34.949 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:34.949 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:34.949 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:34.950 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:34.950 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:34.950 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:34.950 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:34.950 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:34.951 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:34.951 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:34.951 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:34.951 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:34.951 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:34.952 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:34.952 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:34.952 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:34.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:34.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:34.953 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:34.953 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:34.953 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:34.953 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:34.953 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:34.954 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:34.954 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:34.955 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.438188_130.1 2023-10-12T19:08:39.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:39.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:39.957 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:40.000 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:40.000 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:40.050 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:40.051 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:40.051 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:40.051 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:40.051 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:40.052 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:40.052 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:40.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:40.052 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:40.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:40.053 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:40.053 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:40.053 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:40.053 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:08:40.053 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:40.054 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:40.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:40.054 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:08:40.054 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:40.054 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:40.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:40.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:40.055 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:40.055 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:40.055 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:40.056 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:40.056 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:40.056 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:40.056 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:40.056 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:40.057 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:40.057 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:40.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:40.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:40.057 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:40.058 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:40.058 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:40.058 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:40.058 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:40.058 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:40.059 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:40.059 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:40.059 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:40.059 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:40.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:40.060 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:40.060 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:40.060 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:40.060 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:40.060 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:40.061 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:40.061 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:40.061 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:40.061 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:40.061 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:40.062 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:40.062 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:40.062 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:40.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:40.062 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:40.063 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:40.063 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:40.063 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:40.063 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:40.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:08:45.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:45.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:45.068 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:45.111 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:45.111 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:45.163 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:45.164 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:45.164 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:45.164 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:45.164 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:45.164 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:45.165 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:45.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:45.165 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:45.165 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:45.165 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:45.166 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:45.166 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:45.166 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:08:45.166 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:45.167 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:45.167 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:45.167 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:08:45.167 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:45.167 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:45.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:45.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:45.168 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:45.168 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:45.168 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:45.168 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:45.169 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:45.169 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:45.169 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:45.169 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:45.169 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:45.170 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:45.170 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:45.170 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:45.170 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:45.171 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:45.171 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:45.171 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:45.171 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:45.171 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:45.172 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:45.172 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:45.172 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:45.172 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:45.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:45.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:45.173 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:45.173 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:45.173 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:45.174 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:45.174 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:45.174 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:45.174 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:45.174 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:45.175 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:45.175 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:45.175 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:45.175 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:45.175 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:45.176 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:45.176 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:45.176 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:45.176 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:45.176 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:45.179 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.435183_128.1 2023-10-12T19:08:50.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:50.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:50.182 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:50.230 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:50.230 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:50.295 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:50.296 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:50.296 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:50.296 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:50.297 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:50.297 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:50.297 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:50.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:50.297 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:50.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:50.298 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:50.298 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:50.298 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:50.298 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.393673_121.1", 2023-10-12T19:08:50.299 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:50.299 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:50.299 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:50.299 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.393673Z", 2023-10-12T19:08:50.299 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:50.299 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:50.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:50.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:50.300 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:50.300 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:50.300 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:50.301 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:50.301 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:50.301 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:50.301 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:50.301 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:50.301 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:50.302 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:50.302 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:50.302 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:50.302 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:50.302 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:50.303 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:50.303 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:50.303 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:50.303 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:50.303 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:50.304 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:50.304 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:50.304 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:50.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:50.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:50.305 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:50.305 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:50.305 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:50.305 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:50.305 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:50.306 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:50.306 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:50.306 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:50.306 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:50.306 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:50.307 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:50.307 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:50.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:50.307 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:50.307 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:50.308 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:50.308 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:50.308 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:50.309 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.393673_121.1 2023-10-12T19:08:55.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:55.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:08:55.312 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:08:55.354 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:08:55.355 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:08:55.406 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:08:55.406 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:08:55.406 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:08:55.407 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:08:55.407 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:08:55.407 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:08:55.407 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:55.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:55.408 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:08:55.408 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:55.408 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:08:55.408 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:55.408 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:55.409 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T19:08:55.409 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:55.409 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:08:55.409 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:55.410 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T19:08:55.410 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:55.410 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:55.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:55.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:55.411 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:08:55.411 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:55.412 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:55.412 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:08:55.412 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:55.412 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:55.413 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:55.413 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:08:55.413 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:55.413 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:55.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:55.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:55.414 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:08:55.414 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:55.414 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:55.414 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:55.415 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:55.415 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:55.415 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:55.415 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:55.415 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:55.416 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:55.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:55.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:08:55.416 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:08:55.416 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:08:55.416 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:08:55.417 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:08:55.417 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:08:55.417 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:08:55.417 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:08:55.417 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:08:55.418 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:08:55.418 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:55.418 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:55.418 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:08:55.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:08:55.419 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:08:55.419 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:08:55.419 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:08:55.419 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:08:55.419 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:08:55.420 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.473047_134.1 2023-10-12T19:09:00.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:00.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:00.423 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:00.466 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:00.467 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:00.519 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:00.520 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:00.520 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:00.521 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:00.521 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:00.521 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:00.522 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:00.522 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:00.522 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:00.523 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:00.523 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:00.523 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:00.524 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:00.524 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:09:00.524 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:00.524 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:00.525 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:00.525 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:09:00.525 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:00.525 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:00.525 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:00.526 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:00.526 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:00.526 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:00.526 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:00.526 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:00.527 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:00.527 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:00.527 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:00.527 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:00.527 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:00.528 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:00.528 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:00.528 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:00.528 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:00.528 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:00.529 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:00.529 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:00.529 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:00.529 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:00.529 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:00.530 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:00.530 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:00.530 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:00.530 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:00.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:00.531 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:00.531 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:00.531 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:00.531 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:00.532 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:00.532 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:00.532 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:00.532 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:00.532 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:00.533 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:00.533 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:00.533 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:00.533 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:00.533 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:00.533 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:00.534 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:00.534 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:00.534 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:00.535 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.396100_122.1 2023-10-12T19:09:05.537 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... FAIL 2023-10-12T19:09:05.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:05.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:05.539 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:05.580 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:05.580 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:05.614 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T19:09:05.626 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T19:09:05.627 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:05.627 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697137436.143028_223.1", 2023-10-12T19:09:05.627 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T19:03:56.143028Z" 2023-10-12T19:09:05.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:05.628 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:05.628 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:05.628 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T19:09:05.628 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:05.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:05.629 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:05.629 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:09:05.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:05.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:05.630 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:05.630 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:09:05.630 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:05.630 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T19:09:05.631 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T19:09:05.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:05.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:05.633 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:05.713 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:05.713 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:05.763 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:05.763 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:05.763 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:05.764 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:05.764 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:05.764 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:05.764 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:05.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:05.765 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:05.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:05.765 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:05.765 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:05.765 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:05.766 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:09:05.766 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:05.766 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:05.766 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:05.766 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:09:05.767 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:05.767 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:05.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:05.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:05.767 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:05.768 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:05.768 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:05.768 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:05.769 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:05.769 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:05.769 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:05.769 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:05.769 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:05.770 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:05.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:05.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:05.770 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:05.770 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:05.771 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:05.771 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:05.771 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:05.771 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:05.771 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:05.773 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:05.773 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:05.773 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:05.773 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:05.774 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:05.774 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:05.774 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:05.774 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:05.774 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:05.775 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:05.775 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:05.775 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:05.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:05.775 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:05.776 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:05.776 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:05.776 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:05.776 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:05.777 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.410419_126.1 2023-10-12T19:09:10.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:10.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:10.780 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:10.823 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:10.823 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:10.876 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:10.876 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:10.877 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:10.877 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:10.877 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:10.877 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:10.877 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:10.878 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:10.878 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:10.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:10.878 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:10.878 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:10.879 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:10.879 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:09:10.879 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:10.879 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:10.879 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:10.880 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:09:10.880 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:10.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:10.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:10.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:10.881 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:10.881 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:10.881 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:10.881 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:10.881 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:10.882 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:10.882 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:10.882 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:10.882 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:10.882 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:10.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:10.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:10.883 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:10.883 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:10.883 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:10.883 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:10.884 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:10.884 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:10.884 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:10.884 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:10.885 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:10.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:10.885 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:10.885 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:10.885 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:10.886 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:10.886 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:10.886 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:10.886 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:10.886 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:10.887 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:10.887 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:10.887 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:10.887 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:10.887 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:10.888 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:10.888 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:10.888 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:10.888 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:10.888 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:10.889 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:10.889 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:10.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:09:15.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:15.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:15.892 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:15.936 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:15.936 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:15.990 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:15.991 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:15.991 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:15.991 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:15.991 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:15.992 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:15.992 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:15.992 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:15.992 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:15.993 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:15.993 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:15.993 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:15.993 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:15.993 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:09:15.994 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:15.994 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:15.994 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:15.994 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:09:15.994 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:15.995 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:15.995 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:15.995 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:15.995 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:15.995 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:15.996 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:15.996 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:15.996 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:15.996 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:15.997 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:15.997 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:15.997 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:15.997 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:15.997 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:15.998 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:15.998 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:15.998 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:15.998 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:15.998 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:15.999 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:15.999 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:15.999 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:15.999 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:16.004 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:16.005 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.435183_128.1 2023-10-12T19:09:21.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:21.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:21.008 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:21.050 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:21.050 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:21.100 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:21.101 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:21.101 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:21.101 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:21.101 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:21.102 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:21.102 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:21.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:21.102 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:21.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:21.103 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:21.103 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:21.103 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:21.103 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:09:21.103 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:21.104 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:21.104 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:21.104 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:09:21.104 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:21.104 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:21.104 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:21.105 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:21.105 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:21.105 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:21.105 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:21.105 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:21.106 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:21.106 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:21.106 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:21.106 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:21.106 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:21.107 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:21.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:21.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:21.107 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:21.107 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:21.108 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:21.108 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:21.108 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:21.108 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:21.108 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:21.109 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:21.109 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:21.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:21.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:21.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:21.109 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:21.110 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:21.110 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:21.110 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:21.110 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:21.110 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:21.111 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:21.111 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:21.111 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:21.111 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:21.111 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:21.112 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:21.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:21.112 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:21.112 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:21.112 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:21.113 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:21.113 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:21.114 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:09:26.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:26.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:26.116 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:26.161 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:26.162 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:26.216 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:26.217 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:26.217 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:26.217 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:26.217 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:26.217 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:26.218 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:26.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:26.218 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:26.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:26.218 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:26.219 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:26.219 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:26.219 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:09:26.219 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:26.219 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:26.220 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:26.220 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:09:26.220 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:26.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:26.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:26.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:26.221 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:26.221 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:26.221 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:26.221 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:26.222 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:26.222 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:26.222 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:26.222 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:26.222 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:26.223 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:26.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:26.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:26.223 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:26.224 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:26.224 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:26.224 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:26.224 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:26.224 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:26.225 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:26.225 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:26.225 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:26.225 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:26.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:26.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:26.226 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:26.226 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:26.226 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:26.226 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:26.227 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:26.227 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:26.227 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:26.227 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:26.227 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:26.228 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:26.228 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:26.228 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:26.228 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:26.228 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:26.229 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:26.229 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:26.229 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:26.229 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:26.230 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:09:31.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:31.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:31.232 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:31.278 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:31.279 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:31.335 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:31.335 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:31.335 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:31.336 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:31.336 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:31.336 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:31.336 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:31.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:31.337 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:31.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:31.337 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:31.337 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:31.337 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:31.338 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:09:31.338 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:31.338 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:31.338 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:31.338 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:09:31.339 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:31.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:31.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:31.339 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:31.339 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:31.340 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:31.340 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:31.340 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:31.340 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:31.340 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:31.341 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:31.341 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:31.341 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:31.341 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:31.341 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:31.342 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:31.342 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:31.342 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:31.342 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:31.342 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:31.343 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:31.343 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:31.343 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:31.343 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:31.343 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:31.344 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:31.344 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:31.344 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:31.344 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:31.344 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:31.345 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:31.345 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:31.345 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:31.345 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:31.345 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:31.346 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:31.346 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:31.346 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:31.346 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:31.347 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:31.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:31.347 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:31.347 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:31.348 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:31.348 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:31.348 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:31.350 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:09:36.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:36.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:36.352 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:36.400 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:36.400 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:36.457 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:36.457 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:36.457 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:36.457 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:36.458 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:36.458 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:36.458 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:36.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:36.458 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:36.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:36.459 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:36.459 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:36.459 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:36.459 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:09:36.460 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:36.460 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:36.460 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:36.460 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:09:36.460 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:36.461 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:36.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:36.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:36.461 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:36.461 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:36.461 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:36.462 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:36.462 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:36.462 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:36.462 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:36.462 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:36.463 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:36.463 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:36.463 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:36.463 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:36.463 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:36.464 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:36.464 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:36.464 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:36.464 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:36.464 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:36.465 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:36.465 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:36.465 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:36.465 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:36.465 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:36.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:36.466 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:36.466 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:36.466 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:36.466 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:36.466 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:36.467 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:36.467 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:36.467 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:36.467 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:36.467 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:36.468 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:36.468 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:36.468 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:36.468 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:36.468 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:36.469 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:36.469 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:36.469 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:36.470 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.410419_126.1 2023-10-12T19:09:41.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:41.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:41.472 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:41.516 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:41.516 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:41.567 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:41.567 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:41.568 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:41.568 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:41.568 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:41.568 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:41.568 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:41.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:41.569 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:41.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:41.569 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:41.569 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:41.570 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:41.570 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:09:41.570 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:41.570 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:41.570 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:41.571 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:09:41.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:41.571 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:41.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:41.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:41.572 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:41.572 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:41.572 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:41.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:41.572 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:41.573 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:41.573 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:41.573 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:41.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:41.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:41.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:41.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:41.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:41.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:41.574 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:41.575 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:41.575 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:41.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:41.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:41.575 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:41.575 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:41.576 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:41.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:41.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:41.576 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:41.576 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:41.577 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:41.577 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:41.577 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:41.577 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:41.578 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:41.578 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:41.578 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:41.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:41.579 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:41.579 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:41.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:41.579 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:41.579 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:41.580 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:41.580 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:41.580 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:41.581 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.470944_133.1 2023-10-12T19:09:46.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:46.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:46.583 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:46.625 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:46.625 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:46.676 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:46.676 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:46.677 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:46.677 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:46.677 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:46.678 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:46.678 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:46.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:46.679 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:46.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:46.680 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:46.680 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:46.680 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:46.681 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:09:46.681 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:46.681 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:46.682 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:46.682 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:09:46.683 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:46.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:46.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:46.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:46.684 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:46.685 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:46.685 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:46.685 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:46.686 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:46.686 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:46.686 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:46.687 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:46.687 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:46.687 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:46.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:46.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:46.688 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:46.689 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:46.689 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:46.689 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:46.690 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:46.690 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:46.691 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:46.691 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:46.691 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:46.692 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:46.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:46.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:46.693 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:46.693 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:46.694 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:46.694 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:46.694 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:46.695 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:46.695 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:46.695 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:46.696 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:46.696 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:46.697 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:46.697 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:46.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:46.698 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:46.698 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:46.699 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:46.699 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:46.699 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:46.701 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.432502_127.1 2023-10-12T19:09:51.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:51.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:51.705 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:51.747 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:51.748 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:51.798 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:51.798 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:51.798 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:51.798 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:51.799 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:51.799 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:51.799 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:51.799 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:51.799 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:51.800 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:51.800 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:51.800 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:51.800 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:51.800 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470052_132.1", 2023-10-12T19:09:51.801 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:51.801 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:51.801 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:51.801 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470052Z", 2023-10-12T19:09:51.801 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:51.802 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:51.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:51.802 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:51.802 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:51.802 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:51.803 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:51.803 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:51.803 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:51.803 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:51.803 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:51.804 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:51.804 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:51.804 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:51.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:51.804 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:51.805 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:51.805 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:51.805 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:51.805 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:51.805 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:51.805 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:51.806 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:51.806 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:51.806 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:51.806 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:51.806 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:51.807 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:51.807 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:51.807 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:51.807 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:51.807 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:51.808 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:51.808 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:51.808 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:51.808 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:51.808 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:51.809 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:51.809 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:51.809 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:51.809 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:51.809 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:51.810 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:51.810 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:51.810 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:51.810 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:51.811 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.470052_132.1 2023-10-12T19:09:56.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:56.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:09:56.813 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:09:56.856 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:09:56.856 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:09:56.907 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:09:56.908 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:09:56.908 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:09:56.908 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:09:56.908 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:09:56.908 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:09:56.909 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:56.909 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:56.909 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:09:56.909 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:56.909 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:09:56.910 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:56.910 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:56.910 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:09:56.910 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:56.910 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:09:56.911 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:56.911 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:09:56.911 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:56.911 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:56.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:56.912 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:56.912 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:09:56.912 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:56.912 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:56.912 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:09:56.913 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:56.913 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:56.913 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:56.913 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:09:56.913 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:56.913 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:56.914 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:56.914 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:56.914 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:09:56.914 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:56.917 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:56.917 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:56.918 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:56.918 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:56.918 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:56.918 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:56.918 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:56.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:56.919 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:56.919 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:09:56.919 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:09:56.919 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:09:56.919 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:09:56.920 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:09:56.920 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:09:56.920 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:09:56.920 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:09:56.920 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:09:56.921 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:09:56.921 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:56.921 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:56.921 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:09:56.921 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:09:56.921 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:09:56.922 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:09:56.922 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:09:56.922 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:09:56.923 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:09:56.924 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:10:01.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:01.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:01.926 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:01.975 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:01.976 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:02.034 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:02.035 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:02.035 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:02.035 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:02.035 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:02.036 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:02.036 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:02.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:02.036 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:02.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:02.037 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:02.037 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:02.037 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:02.037 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.486122_135.1", 2023-10-12T19:10:02.037 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:02.038 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:02.038 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:02.038 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.486122Z", 2023-10-12T19:10:02.038 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:02.038 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:02.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:02.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:02.039 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:02.039 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:02.039 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:02.039 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:02.041 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:02.041 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:02.041 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:02.042 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:02.042 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:02.042 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:02.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:02.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:02.043 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:02.043 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:02.043 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:02.043 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:02.043 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:02.043 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:02.044 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:02.044 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:02.044 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:02.044 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:02.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:02.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:02.045 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:02.045 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:02.045 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:02.045 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:02.045 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:02.046 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:02.046 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:02.046 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:02.046 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:02.046 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:02.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:02.047 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:02.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:02.047 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:02.047 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:02.048 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:02.048 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:02.048 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:02.049 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.486122_135.1 2023-10-12T19:10:07.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:07.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:07.052 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:07.095 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:07.095 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:07.148 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:07.148 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:10:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:10:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:07.160 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:07.162 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:10:12.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:12.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:12.164 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:12.210 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:12.267 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:12.267 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:12.267 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:12.268 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:12.268 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:12.268 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:12.268 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:12.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:12.269 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:12.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:12.269 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:12.269 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:12.269 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:12.270 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:10:12.270 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:12.270 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:12.270 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:12.270 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:10:12.271 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:12.271 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:12.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:12.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:12.271 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:12.272 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:12.272 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:12.272 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:12.272 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:12.272 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:12.273 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:12.273 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:12.273 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:12.273 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:12.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:12.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:12.274 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:12.274 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:12.274 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:12.275 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:12.275 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:12.275 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:12.275 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:12.275 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:12.276 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:12.276 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:12.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:12.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:12.277 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:12.277 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:12.277 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:12.277 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:12.278 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:12.278 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:12.278 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:12.278 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:12.278 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:12.279 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:12.279 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:12.279 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:12.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:12.279 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:12.279 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:12.280 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:12.280 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:12.280 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:12.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.410419_126.1 2023-10-12T19:10:17.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:17.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:17.284 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:17.326 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:17.327 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:17.377 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:17.378 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:17.378 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:17.378 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:17.378 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:17.378 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:17.379 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:17.379 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:17.379 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:17.379 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:17.379 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:17.380 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:17.380 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:17.380 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470052_132.1", 2023-10-12T19:10:17.380 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:17.380 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:17.380 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:17.381 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470052Z", 2023-10-12T19:10:17.381 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:17.381 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:17.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:17.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:17.382 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:17.382 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:17.382 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:17.382 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:17.382 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:17.383 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:17.383 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:17.383 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:17.383 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:17.383 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:17.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:17.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:17.384 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:17.384 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:17.384 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:17.384 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:17.385 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:17.385 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:17.385 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:17.385 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:17.385 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:17.386 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:17.386 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:17.386 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:17.386 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:17.386 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:17.386 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:17.387 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:17.387 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:17.387 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:17.387 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:17.387 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:17.388 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:17.388 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:17.388 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:17.388 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:17.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:17.388 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:17.389 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:17.389 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:17.389 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:17.389 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:17.390 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.470052_132.1 2023-10-12T19:10:22.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:22.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:22.393 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:22.438 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:22.438 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:22.490 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:22.490 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:22.491 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:22.491 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:22.491 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:22.491 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:22.491 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:22.492 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:22.492 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:22.492 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:22.492 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:22.492 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:22.493 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:22.493 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T19:10:22.493 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:22.493 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:22.493 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:22.494 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T19:10:22.494 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:22.494 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:22.494 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:22.494 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:22.495 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:22.495 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:22.495 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:22.495 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:22.495 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:22.495 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:22.496 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:22.496 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:22.496 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:22.496 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:22.496 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:22.497 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:22.497 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:22.497 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:22.497 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:22.497 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:22.497 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:22.498 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:22.498 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:22.498 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:22.498 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:22.499 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:22.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:22.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:22.499 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:22.500 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:22.500 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:22.500 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:22.500 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:22.500 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:22.501 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:22.501 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:22.501 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:22.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:22.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:22.501 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:22.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:22.502 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:22.502 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:22.502 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:22.502 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:22.503 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:22.504 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.438188_130.1 2023-10-12T19:10:27.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:27.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:27.507 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:27.551 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:27.551 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:27.603 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:27.604 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:27.604 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:27.604 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:27.604 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:27.605 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:27.605 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:27.605 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:27.605 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:27.605 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:27.606 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:27.606 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:27.606 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:27.606 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:10:27.607 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:27.607 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:27.607 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:27.607 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:10:27.608 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:27.608 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:27.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:27.608 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:27.608 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:27.609 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:27.609 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:27.609 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:27.609 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:27.610 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:27.610 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:27.610 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:27.610 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:27.610 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:27.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:27.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:27.611 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:27.611 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:27.611 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:27.612 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:27.612 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:27.612 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:27.612 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:27.612 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:27.613 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:27.613 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:27.613 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:27.613 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:27.613 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:27.614 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:27.614 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:27.614 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:27.614 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:27.614 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:27.615 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:27.615 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:27.615 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:27.615 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:27.615 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:27.616 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:27.616 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:27.616 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:27.616 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:27.616 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:27.616 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:27.617 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:27.618 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.410419_126.1 2023-10-12T19:10:32.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:32.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:32.620 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:32.665 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:32.665 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:32.724 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:32.724 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:32.724 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:32.724 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:32.725 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:32.725 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:32.725 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:32.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:32.725 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:32.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:32.726 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:32.726 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:32.726 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:32.726 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:10:32.727 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:32.727 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:32.727 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:32.727 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:10:32.727 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:32.727 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:32.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:32.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:32.728 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:32.728 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:32.728 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:32.728 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:32.729 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:32.729 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:32.729 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:32.729 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:32.729 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:32.730 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:32.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:32.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:32.730 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:32.730 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:32.730 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:32.731 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:32.731 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:32.731 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:32.731 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:32.731 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:32.732 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:32.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:32.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:32.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:32.732 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:32.732 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:32.733 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:32.733 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:32.733 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:32.733 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:32.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:32.733 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:32.734 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:32.734 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:32.734 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:32.734 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:32.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:32.735 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:32.735 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:32.735 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:32.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:32.735 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:32.736 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:10:37.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:37.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:37.739 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:37.789 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:37.789 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:37.847 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:37.847 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:37.848 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:37.848 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:37.848 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:37.848 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:37.848 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:37.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:37.849 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:37.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:37.849 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:37.849 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:37.850 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:37.850 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:10:37.850 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:37.850 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:37.850 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:37.851 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:10:37.851 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:37.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:37.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:37.851 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:37.852 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:37.852 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:37.852 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:37.852 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:37.852 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:37.853 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:37.853 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:37.853 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:37.853 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:37.853 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:37.854 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:37.854 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:37.854 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:37.854 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:37.854 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:37.855 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:37.855 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:37.855 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:37.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:37.855 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:37.856 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:37.856 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:37.856 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:37.856 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:37.856 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:37.857 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:37.857 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:37.857 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:37.857 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:37.857 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:37.858 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:37.858 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:37.858 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:37.858 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:37.859 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:37.859 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:37.859 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:37.859 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:37.860 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:37.860 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:37.860 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:37.860 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:37.861 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.396100_122.1 2023-10-12T19:10:42.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:42.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:42.864 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:42.906 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:42.906 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:42.961 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:42.962 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:42.962 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:42.962 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:42.964 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:42.964 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:42.964 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:10:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:42.966 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:10:42.966 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:42.966 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:42.970 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:42.970 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:42.970 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:42.970 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:42.971 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:42.971 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:42.971 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:42.971 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:42.971 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:42.972 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:42.972 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:42.972 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:42.972 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:42.973 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:42.973 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:42.973 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:42.973 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:42.973 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:42.973 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:42.974 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:42.974 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:42.974 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:42.974 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:42.975 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:42.975 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:42.975 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:42.975 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:42.976 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:42.976 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:42.976 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:42.976 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:42.977 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:10:47.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:47.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:47.980 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:48.028 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:48.028 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:48.087 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:48.087 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:48.087 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:48.087 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:48.088 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:48.088 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:48.088 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:48.088 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:48.088 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:48.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:48.089 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:48.089 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:48.089 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:48.089 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:10:48.089 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:48.090 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:48.090 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:48.090 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:10:48.090 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:48.090 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:48.091 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:48.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:48.091 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:48.091 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:48.091 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:48.092 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:48.092 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:48.092 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:48.092 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:48.092 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:48.092 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:48.093 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:48.093 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:48.093 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:48.093 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:48.093 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:48.094 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:48.094 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:48.094 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:48.094 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:48.094 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:48.095 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:48.095 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:48.095 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:48.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:48.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:48.095 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:48.096 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:48.096 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:48.096 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:48.096 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:48.096 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:48.097 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:48.097 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:48.097 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:48.097 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:48.097 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:48.098 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:48.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:48.098 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:48.098 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:48.098 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:48.099 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:48.099 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:48.100 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.410419_126.1 2023-10-12T19:10:53.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:53.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:53.102 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:53.147 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:53.147 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:53.202 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:53.203 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:53.203 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:53.203 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:53.203 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:53.203 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:53.204 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:53.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:53.204 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:53.204 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:53.204 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:53.205 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:53.205 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:53.205 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:10:53.205 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:53.205 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:53.206 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:53.206 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:10:53.206 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:53.206 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:53.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:53.207 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:53.207 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:53.207 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:53.207 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:53.207 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:53.211 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:53.211 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:53.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:53.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:53.211 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:53.212 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:53.212 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:53.212 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:53.212 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:53.212 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:53.215 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:53.216 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:10:58.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:58.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:10:58.218 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:10:58.268 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:10:58.269 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:10:58.327 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:10:58.327 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:10:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:10:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:10:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:10:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:10:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:10:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:10:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:58.330 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:58.330 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.764432_172.1", 2023-10-12T19:10:58.330 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:58.330 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:10:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.764432Z", 2023-10-12T19:10:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:58.332 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:10:58.332 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:58.332 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:58.332 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:10:58.332 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:10:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:10:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:58.335 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:58.335 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:58.335 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:58.335 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:10:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:10:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:10:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:10:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:10:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:10:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:10:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:10:58.338 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:10:58.338 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:10:58.338 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:58.338 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:10:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:10:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:10:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:10:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:10:58.340 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:10:58.340 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:10:58.341 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.764432_172.1 2023-10-12T19:11:03.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:03.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:03.343 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:03.387 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:03.387 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:03.438 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:03.439 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:03.439 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:03.439 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:03.439 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:03.440 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:03.440 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:03.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:03.440 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:03.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:03.440 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:03.441 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:03.441 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:03.441 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:11:03.441 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:03.441 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:03.442 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:03.442 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:11:03.442 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:03.442 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:03.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:03.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:03.443 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:03.443 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:03.444 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:03.444 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:03.444 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:03.444 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:03.444 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:03.445 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:03.445 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:03.445 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:03.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:03.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:03.445 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:03.446 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:03.446 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:03.446 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:03.446 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:03.446 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:03.447 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:03.447 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:03.447 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:03.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:03.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:03.448 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:03.448 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:03.448 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:03.448 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:03.448 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:03.449 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:03.449 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:03.449 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:03.449 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:03.449 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:03.450 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:03.450 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:03.450 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:03.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:03.450 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:03.451 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:03.451 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:03.451 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:03.451 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:03.452 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:11:08.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:08.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:08.455 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:08.496 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:08.497 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:08.549 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:08.550 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:08.550 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:08.550 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:08.550 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:08.551 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:08.551 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:08.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:08.551 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:08.551 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:08.552 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:08.552 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:08.552 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:08.552 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:11:08.552 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:08.553 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:08.553 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:08.553 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:11:08.553 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:08.553 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:08.553 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:08.554 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:08.554 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:08.554 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:08.554 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:08.554 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:08.555 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:08.555 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:08.555 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:08.555 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:08.555 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:08.556 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:08.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:08.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:08.556 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:08.557 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:08.557 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:08.557 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:08.558 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:08.558 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:08.558 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:08.558 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:08.559 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:08.559 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:08.559 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:08.559 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:08.560 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:08.560 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:08.560 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:08.560 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:08.560 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:08.561 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:08.561 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:08.561 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:08.561 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:08.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:08.562 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:08.562 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:08.562 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:08.562 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:08.563 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:08.563 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:08.563 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:08.563 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:08.564 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.432502_127.1 2023-10-12T19:11:13.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:13.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:13.567 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:13.610 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:13.610 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:13.661 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:13.661 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:13.661 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:13.661 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:13.662 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:13.662 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:13.662 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:13.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:13.662 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:13.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:13.663 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:13.663 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:13.663 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:13.663 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470052_132.1", 2023-10-12T19:11:13.664 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:13.664 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:13.664 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:13.664 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470052Z", 2023-10-12T19:11:13.664 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:13.664 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:13.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:13.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:13.665 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:13.665 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:13.665 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:13.666 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:13.666 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:13.666 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:13.666 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:13.666 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:13.667 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:13.667 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:13.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:13.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:13.667 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:13.668 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:13.668 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:13.668 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:13.668 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:13.668 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:13.669 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:13.669 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:13.669 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:13.669 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:13.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:13.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:13.670 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:13.670 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:13.670 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:13.670 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:13.670 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:13.671 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:13.671 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:13.671 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:13.671 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:13.671 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:13.672 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:13.672 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:13.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:13.672 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:13.672 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:13.673 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:13.673 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:13.673 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:13.674 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.470052_132.1 2023-10-12T19:11:18.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:18.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:18.677 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:18.720 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:18.721 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:18.772 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:18.772 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:18.772 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:18.773 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:18.773 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:18.773 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:18.773 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:18.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:18.774 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:18.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:18.775 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:18.775 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:18.775 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:18.775 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.554239_141.1", 2023-10-12T19:11:18.775 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:18.775 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:18.776 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:18.776 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.554239Z", 2023-10-12T19:11:18.776 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:18.776 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:18.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:18.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:18.777 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:18.777 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:18.777 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:18.777 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:18.778 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:18.778 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:18.778 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:18.778 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:18.778 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:18.779 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:18.779 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:18.779 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:18.779 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:18.779 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:18.780 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:18.780 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:18.780 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:18.780 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:18.780 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:18.781 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:18.781 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:18.781 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:18.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:18.782 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:18.782 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:18.782 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:18.782 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:18.783 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:18.783 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:18.783 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:18.783 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:18.783 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:18.784 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:18.784 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:18.784 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:18.784 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:18.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:18.785 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:18.785 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:18.785 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:18.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:18.785 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:18.786 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.554239_141.1 2023-10-12T19:11:23.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:23.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:23.789 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:23.831 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:23.832 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:23.882 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:23.882 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:23.883 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:23.883 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:23.883 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:23.883 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:23.883 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:23.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:23.884 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:23.884 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:23.884 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:23.884 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:23.885 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:23.885 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:11:23.885 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:23.885 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:23.885 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:23.885 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:11:23.886 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:23.886 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:23.886 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:23.886 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:23.887 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:23.887 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:23.887 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:23.887 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:23.887 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:23.888 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:23.888 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:23.888 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:23.888 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:23.888 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:23.889 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:23.889 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:23.889 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:23.889 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:23.889 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:23.890 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:23.890 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:23.890 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:23.890 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:23.890 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:23.891 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:23.891 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:23.891 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:23.891 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:23.891 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:23.892 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:23.892 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:23.892 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:23.892 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:23.892 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:23.893 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:23.893 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:23.893 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:23.893 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:23.893 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:23.893 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:23.894 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:23.894 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:23.894 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:23.894 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:23.894 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:23.895 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:23.896 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:11:28.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:28.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:28.898 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:28.944 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:28.945 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:29.000 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:29.000 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:29.000 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:29.000 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:29.001 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:29.001 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:29.001 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:29.001 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:29.001 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:29.002 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:29.002 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:29.002 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:29.002 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:29.002 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:11:29.003 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:29.003 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:29.003 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:29.003 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:11:29.003 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:29.004 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:29.004 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:29.004 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:29.004 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:29.004 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:29.005 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:29.005 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:29.005 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:29.005 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:29.005 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:29.006 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:29.006 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:29.006 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:29.006 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:29.006 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:29.007 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:29.007 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:29.007 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:29.007 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:29.007 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:29.008 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:29.008 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:29.008 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:29.008 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:29.008 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:29.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:29.009 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:29.009 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:29.009 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:29.009 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:29.012 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:29.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.435183_128.1 2023-10-12T19:11:34.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:34.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:34.016 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:34.058 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:34.058 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:34.109 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:34.110 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:34.111 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:34.111 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:34.111 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:34.111 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:34.111 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:34.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:34.112 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:34.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:34.112 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:34.112 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:34.113 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:34.113 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:11:34.113 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:34.113 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:34.113 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:34.114 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:11:34.114 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:34.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:34.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:34.114 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:34.115 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:34.115 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:34.115 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:34.115 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:34.115 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:34.116 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:34.116 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:34.116 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:34.116 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:34.116 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:34.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:34.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:34.117 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:34.117 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:34.117 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:34.118 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:34.118 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:34.118 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:34.118 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:34.118 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:34.119 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:34.119 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:34.119 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:34.119 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:34.119 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:34.120 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:34.120 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:34.120 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:34.120 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:34.120 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:34.121 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:34.121 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:34.121 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:34.121 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:34.121 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:34.121 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:34.122 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:34.122 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:34.122 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:34.122 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:34.123 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:34.123 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:34.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:11:39.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:39.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:39.126 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:39.168 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:39.168 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:39.220 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:39.220 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:39.220 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:39.221 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:39.221 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:39.221 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:39.221 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:39.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:39.222 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:39.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:39.222 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:39.222 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:39.223 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:39.223 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:11:39.223 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:39.223 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:39.223 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:39.224 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:11:39.224 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:39.224 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:39.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:39.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:39.224 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:39.225 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:39.225 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:39.225 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:39.225 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:39.225 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:39.226 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:39.226 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:39.226 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:39.226 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:39.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:39.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:39.227 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:39.227 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:39.227 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:39.227 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:39.227 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:39.232 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:39.233 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:11:44.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:44.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:44.236 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:44.279 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:44.279 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:44.331 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:44.331 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:44.331 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:44.331 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:44.332 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:44.332 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:44.332 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:44.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:44.332 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:44.333 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:44.333 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:44.333 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:44.333 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:44.334 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:11:44.334 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:44.334 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:44.334 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:44.334 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:11:44.335 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:44.335 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:44.335 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:44.335 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:44.335 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:44.335 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:44.336 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:44.336 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:44.336 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:44.336 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:44.336 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:44.337 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:44.337 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:44.337 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:44.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:44.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:44.338 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:44.338 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:44.338 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:44.338 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:44.338 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:44.339 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:44.339 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:44.339 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:44.339 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:44.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:44.340 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:44.340 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:44.340 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:44.340 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:44.340 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:44.340 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:44.341 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:44.341 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:44.341 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:44.341 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:44.341 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:44.342 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:44.342 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:44.342 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:44.342 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:44.342 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:44.343 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:44.343 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:44.343 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:44.343 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:44.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:11:49.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:49.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:49.347 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:49.394 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:49.394 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:49.458 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:49.458 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:49.459 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:49.459 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:49.459 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:49.459 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:49.459 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:49.460 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:49.460 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:49.460 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:49.460 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:49.460 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:49.461 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:49.461 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:11:49.461 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:49.461 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:49.461 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:49.462 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:11:49.462 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:49.462 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:49.462 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:49.462 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:49.463 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:49.463 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:49.463 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:49.463 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:49.463 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:49.464 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:49.464 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:49.464 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:49.464 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:49.464 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:49.465 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:49.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:49.465 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:49.465 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:49.465 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:49.466 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:49.466 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:49.466 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:49.466 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:49.466 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:49.467 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:49.467 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:49.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:49.467 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:49.467 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:49.468 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:49.468 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:49.468 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:49.468 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:49.468 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:49.468 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:49.469 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:49.469 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:49.469 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:49.469 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:49.469 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:49.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:49.470 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:49.470 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:49.470 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:49.470 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:49.471 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:49.472 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:11:54.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:54.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:54.474 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:54.516 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:54.516 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:54.569 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:54.570 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:54.570 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:54.570 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:54.570 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:54.571 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:54.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:54.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:54.571 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:54.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:54.572 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:54.572 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:54.572 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:54.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:11:54.572 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:54.573 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:54.573 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:54.573 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:11:54.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:54.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:54.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:54.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:54.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:54.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:54.574 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:54.574 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:54.575 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:54.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:54.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:54.575 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:54.575 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:54.576 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:54.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:54.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:54.576 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:54.576 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:54.577 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:54.577 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:54.577 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:54.577 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:54.577 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:54.578 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:54.578 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:54.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:54.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:54.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:54.579 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:54.579 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:54.579 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:54.579 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:54.579 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:54.579 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:54.580 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:54.580 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:54.580 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:54.580 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:54.580 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:54.581 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:54.581 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:54.581 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:54.581 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:54.581 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:54.581 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:54.582 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:54.582 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:11:59.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:59.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:11:59.585 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:11:59.627 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:11:59.627 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:11:59.678 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:11:59.678 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:11:59.678 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:11:59.678 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:11:59.679 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:11:59.679 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:11:59.679 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:59.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:59.679 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:11:59.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:59.680 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:11:59.680 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:59.680 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:59.680 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:11:59.681 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:59.681 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:11:59.681 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:59.681 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:11:59.681 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:59.682 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:59.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:59.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:59.682 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:11:59.682 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:59.683 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:59.683 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:11:59.683 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:59.683 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:59.683 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:59.684 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:11:59.684 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:59.684 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:59.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:59.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:59.685 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:11:59.685 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:59.685 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:59.685 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:59.685 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:59.686 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:59.686 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:59.686 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:59.686 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:59.686 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:59.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:59.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:11:59.687 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:11:59.687 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:11:59.687 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:11:59.688 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:11:59.688 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:11:59.688 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:11:59.688 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:11:59.688 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:11:59.688 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:11:59.689 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:59.689 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:59.689 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:11:59.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:11:59.689 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:11:59.689 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:11:59.690 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:11:59.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:11:59.690 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:11:59.691 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.467824_131.1 2023-10-12T19:12:04.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:04.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:04.693 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:04.743 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:04.744 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:04.801 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:04.801 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:04.802 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:04.802 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:04.802 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:04.802 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:04.802 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:04.803 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:04.803 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:04.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:04.803 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:04.803 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:04.804 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:04.804 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:12:04.804 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:04.804 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:04.804 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:04.805 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:12:04.805 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:04.805 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:04.805 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:04.805 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:04.805 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:04.806 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:04.806 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:04.806 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:04.807 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:04.807 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:04.807 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:04.807 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:04.807 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:04.808 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:04.808 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:04.808 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:04.808 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:04.808 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:04.808 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:04.809 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:04.809 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:04.809 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:04.809 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:04.809 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:04.810 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:04.810 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:04.810 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:04.810 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:04.810 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:04.811 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:04.811 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:04.811 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:04.811 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:04.811 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:04.811 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:04.812 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:04.812 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:04.812 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:04.812 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:04.812 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:04.813 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:04.813 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:04.813 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:04.813 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:04.813 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:04.814 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:04.815 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.467824_131.1 2023-10-12T19:12:09.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:09.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:09.817 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:09.859 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:09.859 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:09.911 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:09.911 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:09.912 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:09.912 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:09.912 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:09.912 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:09.913 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:09.913 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:09.913 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:09.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:09.913 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:09.913 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:09.914 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:09.914 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:12:09.914 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:09.914 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:09.914 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:09.915 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:12:09.915 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:09.915 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:09.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:09.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:09.916 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:09.916 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:09.916 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:09.916 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:09.916 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:09.917 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:09.917 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:09.917 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:09.917 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:09.917 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:09.918 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:09.918 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:09.918 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:09.918 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:09.918 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:09.919 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:09.919 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:09.919 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:09.919 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:09.919 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:09.920 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:09.920 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:09.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:09.920 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:09.920 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:09.921 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:09.921 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:09.921 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:09.921 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:09.921 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:09.922 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:09.922 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:09.922 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:09.922 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:09.922 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:09.923 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:09.923 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:09.923 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:09.923 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:09.923 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:09.924 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:09.924 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:09.925 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.470944_133.1 2023-10-12T19:12:14.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:14.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:14.927 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:14.969 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:14.969 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:15.020 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:15.020 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:15.020 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:15.020 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:15.021 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:15.021 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:15.021 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:15.021 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:15.021 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:15.022 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:15.022 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:15.022 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:15.022 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:15.022 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:12:15.023 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:15.023 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:15.023 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:15.023 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:12:15.023 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:15.024 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:15.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:15.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:15.024 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:15.024 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:15.025 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:15.025 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:15.025 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:15.025 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:15.025 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:15.032 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:15.032 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:15.033 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:12:20.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:20.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:20.035 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:20.079 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:20.080 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:20.132 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:20.132 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:20.132 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:20.133 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:20.133 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:20.133 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:20.133 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:20.133 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:20.134 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:20.134 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:20.134 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:20.134 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:20.134 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:20.135 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:12:20.135 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:20.135 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:20.135 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:20.136 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:12:20.136 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:20.136 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:20.136 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:20.136 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:20.136 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:20.137 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:20.137 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:20.137 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:20.137 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:20.137 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:20.138 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:20.138 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:20.138 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:20.138 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:20.138 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:20.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:20.139 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:20.139 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:20.139 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:20.139 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:20.140 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:20.140 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:20.140 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:20.140 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:20.140 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:20.140 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:20.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:20.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:20.141 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:20.141 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:20.141 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:20.142 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:20.142 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:20.142 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:20.142 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:20.142 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:20.143 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:20.143 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:20.143 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:20.143 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:20.143 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:20.143 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:20.144 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:20.144 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:20.144 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:20.144 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:20.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.432502_127.1 2023-10-12T19:12:25.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:25.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:25.148 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:25.191 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:25.191 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:25.247 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:25.247 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:25.247 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:25.247 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:25.248 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:25.248 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:25.248 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:25.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:25.248 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:25.249 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:25.249 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:25.249 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:25.249 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:25.249 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.554239_141.1", 2023-10-12T19:12:25.250 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:25.250 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:25.250 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:25.250 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.554239Z", 2023-10-12T19:12:25.250 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:25.251 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:25.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:25.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:25.251 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:25.251 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:25.251 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:25.252 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:25.252 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:25.252 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:25.252 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:25.253 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:25.253 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:25.253 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:25.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:25.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:25.254 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:25.254 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:25.254 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:25.254 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:25.254 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:25.255 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:25.255 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:25.255 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:25.255 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:25.255 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:25.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:25.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:25.256 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:25.256 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:25.256 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:25.256 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:25.257 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:25.257 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:25.257 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:25.257 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:25.257 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:25.258 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:25.258 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:25.258 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:25.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:25.258 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:25.259 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:25.259 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:25.259 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:25.259 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:25.260 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.554239_141.1 2023-10-12T19:12:30.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:30.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:30.263 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:30.306 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:30.306 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:30.357 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:30.357 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:30.358 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:30.358 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:30.358 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:30.358 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:30.359 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:30.359 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:30.359 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:30.359 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:30.359 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:30.360 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:30.360 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:30.360 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:12:30.360 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:30.360 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:30.361 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:30.361 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:12:30.361 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:30.361 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:30.361 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:30.361 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:30.362 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:30.362 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:30.362 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:30.362 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:30.362 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:30.363 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:30.363 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:30.363 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:30.363 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:30.363 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:30.364 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:30.364 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:30.364 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:30.364 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:30.364 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:30.365 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:30.365 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:30.365 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:30.365 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:30.366 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:30.366 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:30.366 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:30.366 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:30.366 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:30.367 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:30.367 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:30.367 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:30.367 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:30.367 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:30.368 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:30.368 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:30.368 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:30.368 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:30.368 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:30.369 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:30.369 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:30.369 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:30.369 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:30.369 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:30.370 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:30.370 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:30.370 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:30.371 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.435183_128.1 2023-10-12T19:12:35.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:35.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:35.374 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:35.416 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:35.416 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:35.710 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:35.710 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:35.710 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:35.711 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:35.711 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:35.711 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:35.711 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:35.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:35.712 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:35.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:35.712 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:35.712 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:35.713 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:35.713 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:12:35.713 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:35.714 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:35.714 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:35.714 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:12:35.714 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:35.714 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:35.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:35.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:35.715 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:35.715 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:35.715 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:35.716 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:35.716 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:35.716 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:35.716 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:35.716 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:35.717 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:35.717 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:35.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:35.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:35.717 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:35.718 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:35.718 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:35.718 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:35.718 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:35.718 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:35.719 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:35.719 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:35.719 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:35.719 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:35.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:35.721 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:35.721 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:35.722 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:35.722 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:35.722 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:35.722 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:35.723 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:35.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:35.723 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:35.723 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:35.723 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:35.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:35.724 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:35.725 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:12:40.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:40.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:40.728 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:40.771 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:40.771 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:40.822 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:40.822 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:40.822 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:40.823 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:40.823 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:40.823 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:40.823 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:40.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:40.824 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:40.824 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:40.824 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:40.824 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:40.824 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:40.825 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:12:40.825 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:40.825 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:40.825 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:40.825 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:12:40.826 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:40.826 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:40.826 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:40.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:40.826 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:40.827 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:40.827 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:40.827 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:40.827 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:40.827 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:40.828 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:40.828 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:40.828 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:40.828 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:40.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:40.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:40.829 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:40.829 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:40.829 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:40.829 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:40.830 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:40.830 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:40.830 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:40.830 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:40.830 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:40.831 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:40.831 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:40.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:40.831 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:40.831 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:40.832 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:40.832 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:40.832 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:40.832 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:40.832 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:40.833 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:40.833 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:40.833 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:40.833 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:40.833 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:40.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:40.834 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:40.834 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:40.834 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:40.834 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:40.835 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:40.836 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.432502_127.1 2023-10-12T19:12:45.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:45.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:45.839 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:45.882 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:45.883 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:45.934 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:45.934 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:45.934 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:45.934 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:45.935 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:45.935 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:45.935 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:45.935 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:45.936 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:45.936 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:45.936 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:45.936 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:45.936 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:45.937 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:12:45.937 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:45.937 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:45.937 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:45.937 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:12:45.938 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:45.938 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:45.938 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:45.938 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:45.939 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:45.939 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:45.939 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:45.939 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:45.939 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:45.939 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:45.940 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:45.940 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:45.940 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:45.940 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:45.941 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:45.941 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:45.941 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:45.941 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:45.941 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:45.942 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:45.942 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:45.942 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:45.942 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:45.943 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:45.943 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:45.943 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:45.943 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:45.944 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:45.944 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:45.945 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:45.945 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:45.945 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:45.946 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:45.946 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:45.946 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:45.947 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:45.947 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:45.947 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:45.947 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:45.948 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:45.948 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:45.948 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:45.949 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:45.949 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:45.949 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:45.949 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:45.951 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:12:50.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:50.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:50.954 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:51.003 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:51.003 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:51.063 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:51.063 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:51.063 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:51.064 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:51.064 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:51.064 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:51.064 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:51.064 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:51.065 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:51.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:51.065 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:51.065 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:51.066 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:51.066 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:12:51.066 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:51.066 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:51.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:51.067 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:12:51.067 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:51.067 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:51.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:51.067 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:51.068 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:51.068 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:51.068 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:51.068 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:51.068 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:51.069 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:51.069 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:51.069 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:51.069 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:51.069 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:51.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:51.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:51.070 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:51.070 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:51.070 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:51.070 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:51.071 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:51.071 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:51.071 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:51.071 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:51.071 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:51.072 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:51.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:51.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:51.072 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:51.072 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:51.073 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:51.073 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:51.073 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:51.073 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:51.073 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:51.074 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:51.074 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:51.074 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:51.074 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:51.074 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:51.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:51.075 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:51.075 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:51.075 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:51.075 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:51.075 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:51.076 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.396100_122.1 2023-10-12T19:12:56.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:56.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:12:56.079 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:12:56.127 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:12:56.127 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:12:56.184 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:12:56.185 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:12:56.185 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:12:56.185 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:12:56.185 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:12:56.185 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:12:56.186 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:56.186 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:56.186 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:12:56.186 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:56.186 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:12:56.187 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:56.187 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:56.187 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:12:56.188 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:56.188 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:12:56.188 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:56.188 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:12:56.189 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:56.189 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:56.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:56.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:56.189 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:12:56.190 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:56.190 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:56.190 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:12:56.190 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:56.190 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:56.191 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:56.191 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:12:56.191 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:56.191 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:56.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:56.192 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:56.192 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:12:56.192 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:56.192 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:56.192 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:56.193 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:56.193 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:56.193 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:56.193 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:56.193 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:56.194 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:56.194 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:56.194 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:12:56.194 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:12:56.194 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:12:56.195 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:12:56.195 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:12:56.195 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:12:56.195 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:12:56.195 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:12:56.196 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:12:56.196 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:12:56.196 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:56.196 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:56.196 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:12:56.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:12:56.197 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:12:56.197 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:12:56.197 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:12:56.197 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:12:56.197 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:12:56.199 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:13:01.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:01.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:01.201 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:01.242 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:01.243 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:01.295 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:01.295 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:01.295 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:01.295 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:13:01.298 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:01.298 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:01.298 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:01.298 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:13:01.298 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:01.299 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:01.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:01.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:01.299 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:01.299 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:01.300 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:01.300 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:01.300 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:01.300 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:01.300 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:01.300 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:01.301 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:01.301 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:01.301 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:01.301 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:01.301 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:01.302 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:01.302 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:01.302 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:01.302 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:01.302 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:01.303 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:01.303 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:01.303 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:01.303 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:01.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:01.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:01.304 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:01.304 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:01.304 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:01.305 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:01.305 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:01.305 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:01.305 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:01.305 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:01.306 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:01.306 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:01.306 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:01.306 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:01.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:01.307 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:01.307 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:01.307 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:01.307 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:01.307 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:01.308 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.432502_127.1 2023-10-12T19:13:06.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:06.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:06.310 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:06.354 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:06.354 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:06.407 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:06.408 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:06.408 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:06.408 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:06.409 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:06.409 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:06.409 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:06.409 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:06.409 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:06.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:06.410 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:06.410 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:06.410 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:06.410 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:13:06.411 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:06.411 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:06.411 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:06.411 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:13:06.411 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:06.412 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:06.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:06.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:06.412 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:06.412 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:06.413 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:06.413 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:06.413 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:06.413 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:06.413 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:06.414 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:06.414 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:06.414 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:06.414 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:06.414 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:06.415 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:06.415 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:06.415 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:06.415 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:06.415 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:06.415 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:06.416 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:06.416 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:06.416 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:06.416 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:06.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:06.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:06.417 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:06.417 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:06.417 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:06.417 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:06.418 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:06.418 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:06.418 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:06.418 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:06.418 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:06.419 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:06.419 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:06.419 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:06.419 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:06.419 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:06.420 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:06.420 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:06.420 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:06.420 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:06.421 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:13:11.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:11.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:11.424 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:11.466 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:11.466 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:11.525 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:11.525 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:11.525 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:11.525 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:11.526 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:11.526 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:11.526 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:11.526 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:11.527 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:11.527 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:11.527 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:11.527 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:11.527 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:11.527 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:13:11.528 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:11.528 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:11.528 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:11.528 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:13:11.528 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:11.529 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:11.529 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:11.529 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:11.529 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:11.529 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:11.530 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:11.530 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:11.530 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:11.530 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:11.530 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:11.531 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:11.531 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:11.531 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:11.531 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:11.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:11.531 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:11.532 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:11.532 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:11.532 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:11.532 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:11.532 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:11.533 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:11.533 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:11.533 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:11.533 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:11.533 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:11.534 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:11.534 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:11.534 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:11.534 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:11.534 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:11.535 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:11.535 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:11.535 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:11.535 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:11.535 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:11.535 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:11.536 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:11.536 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:11.536 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:11.536 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:11.536 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:11.537 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:11.537 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:11.537 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:11.538 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.432502_127.1 2023-10-12T19:13:16.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:16.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:16.540 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:16.584 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:16.584 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:16.635 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:16.635 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:16.635 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.396100_122.1", 2023-10-12T19:13:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.396100Z", 2023-10-12T19:13:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:16.640 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:16.640 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:16.640 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:16.640 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:16.640 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:16.641 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:16.641 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:16.641 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:16.641 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:16.641 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:16.642 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:16.642 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:16.642 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:16.642 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:16.642 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:16.643 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:16.643 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:16.643 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:16.643 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:16.643 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:16.643 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:16.644 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:16.644 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:16.644 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:16.644 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:16.645 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:16.645 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:16.645 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:16.645 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:16.645 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:16.646 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:16.646 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:16.646 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:16.646 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:16.646 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:16.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:16.647 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:16.647 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:16.647 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:16.647 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:16.647 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:16.649 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.396100_122.1 2023-10-12T19:13:21.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:21.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:21.652 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:21.695 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:21.695 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:21.748 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:21.748 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:13:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:13:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:21.753 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:21.753 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:21.753 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:21.753 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:21.753 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:21.754 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:21.754 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:21.754 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:21.754 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:21.755 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:21.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:21.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:21.755 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:21.756 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:21.756 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:21.756 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:21.756 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:21.757 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:21.757 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:21.757 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:21.757 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:21.757 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:21.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:21.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:21.758 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:21.758 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:21.759 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:21.759 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:21.759 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:21.759 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:21.759 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:21.760 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:21.760 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:21.760 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:21.760 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:21.760 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:21.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:21.761 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:21.761 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:21.761 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:21.761 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:21.762 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:21.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:13:26.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:26.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:26.765 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:26.806 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:26.807 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:26.857 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:26.858 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:26.858 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:26.858 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:26.858 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:26.859 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:26.859 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:26.859 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:26.859 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:26.859 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:26.860 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:26.860 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:26.860 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:26.860 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:13:26.860 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:26.861 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:26.861 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:26.861 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:13:26.861 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:26.861 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:26.861 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:26.862 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:26.862 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:26.862 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:26.862 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:26.863 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:26.863 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:26.863 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:26.863 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:26.863 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:26.863 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:26.864 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:26.864 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:26.864 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:26.864 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:26.864 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:26.865 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:26.865 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:26.865 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:26.865 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:26.865 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:26.866 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:26.866 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:26.866 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:26.866 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:26.866 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:26.867 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:26.867 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:26.867 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:26.867 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:26.867 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:26.867 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:26.868 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:26.868 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:26.868 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:26.868 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:26.868 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:26.869 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:26.869 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:26.869 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:26.869 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:26.870 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:26.870 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:26.871 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:26.872 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.436071_129.1 2023-10-12T19:13:31.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:31.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:31.874 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:31.918 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:31.918 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:31.972 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:31.972 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:31.973 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:31.973 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:31.973 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:31.973 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:31.973 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:31.974 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:31.974 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:31.974 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:31.974 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:31.975 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:31.975 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:31.975 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:13:31.975 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:31.975 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:31.976 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:31.976 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:13:31.976 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:31.976 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:31.976 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:31.977 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:31.977 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:31.977 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:31.977 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:31.977 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:31.978 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:31.978 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:31.978 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:31.978 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:31.979 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:31.979 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:31.979 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:31.979 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:31.979 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:31.979 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:31.980 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:31.980 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:31.980 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:31.980 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:31.981 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:31.981 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:31.981 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:31.981 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:31.982 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:31.982 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:31.982 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:31.982 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:31.982 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:31.983 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:31.983 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:31.983 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:31.983 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:31.983 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:31.983 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:31.984 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:31.984 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:31.984 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:31.984 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:31.984 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:31.985 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:31.985 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:31.985 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:31.985 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:31.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:13:36.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:36.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:36.989 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:37.033 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:37.033 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:37.086 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:37.086 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:37.086 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:37.087 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:37.087 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:37.087 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:37.087 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:37.087 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:37.089 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:37.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:37.089 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:37.089 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:37.090 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:37.090 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:13:37.090 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:37.090 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:37.091 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:37.091 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:13:37.091 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:37.091 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:37.091 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:37.092 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:37.092 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:37.092 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:37.092 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:37.092 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:37.093 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:37.093 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:37.093 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:37.093 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:37.093 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:37.093 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:37.094 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:37.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:37.094 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:37.094 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:37.094 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:37.095 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:37.095 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:37.095 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:37.095 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:37.095 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:37.096 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:37.096 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:37.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:37.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:37.096 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:37.097 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:37.097 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:37.097 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:37.097 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:37.097 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:37.098 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:37.098 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:37.098 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:37.098 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:37.098 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:37.099 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:37.099 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:37.099 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:37.099 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:37.099 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:37.099 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:37.100 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:37.101 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.407338_124.1 2023-10-12T19:13:42.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:42.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:42.104 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:42.146 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:42.147 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:42.199 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:42.200 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:42.200 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:42.200 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:42.201 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:42.201 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:42.201 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:42.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:42.201 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:42.201 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:42.202 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:42.202 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:42.202 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:42.202 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:13:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:13:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:42.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:42.204 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:42.204 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:42.204 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:42.207 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:42.207 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:42.207 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:42.207 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:42.207 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:42.212 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:42.212 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:42.212 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:42.212 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:42.213 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.432502_127.1 2023-10-12T19:13:47.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:47.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:47.216 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:47.259 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:47.259 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:47.312 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:47.312 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:47.312 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:47.312 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:47.313 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:47.313 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:47.313 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:47.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:47.313 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:47.314 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:47.314 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:47.314 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:47.314 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:47.315 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:13:47.315 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:47.315 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:47.315 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:47.315 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:13:47.315 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:47.316 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:47.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:47.316 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:47.316 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:47.316 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:47.317 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:47.317 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:47.317 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:47.317 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:47.317 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:47.317 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:47.318 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:47.318 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:47.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:47.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:47.318 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:47.319 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:47.319 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:47.319 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:47.319 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:47.319 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:47.319 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:47.320 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:47.320 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:47.320 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:47.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:47.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:47.321 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:47.321 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:47.321 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:47.321 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:47.321 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:47.321 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:47.322 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:47.322 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:47.322 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:47.322 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:47.322 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:47.323 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:47.323 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:47.323 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:47.323 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:47.323 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:47.323 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:47.324 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:47.325 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.470944_133.1 2023-10-12T19:13:52.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:52.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:52.327 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:52.369 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:52.370 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:52.422 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:52.423 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:52.423 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:52.423 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:52.423 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:52.424 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:52.424 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:52.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:52.424 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:52.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:52.425 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:52.425 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:52.425 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:52.425 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:13:52.425 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:52.426 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:52.426 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:52.426 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:13:52.426 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:52.426 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:52.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:52.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:52.427 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:52.427 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:52.427 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:52.428 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:52.428 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:52.428 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:52.428 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:52.428 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:52.429 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:52.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:52.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:52.429 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:52.429 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:52.430 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:52.430 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:52.430 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:52.430 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:52.430 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:52.431 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:52.431 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:52.431 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:52.431 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:52.431 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:52.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:52.432 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:52.432 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:52.432 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:52.432 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:52.433 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:52.433 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:52.433 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:52.433 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:52.434 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:52.434 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:52.434 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:52.434 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:52.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:52.434 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:52.435 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:52.435 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:52.435 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:52.435 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:52.436 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.404891_123.1 2023-10-12T19:13:57.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:57.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:13:57.439 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:13:57.481 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:13:57.482 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:13:57.533 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:13:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:13:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:13:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:13:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:13:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:13:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:13:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:13:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:13:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:13:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:13:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:13:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:13:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:13:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:57.540 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:57.540 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:57.540 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:13:57.540 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:57.540 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:57.540 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:13:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:13:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:13:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:13:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:13:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:13:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:13:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:13:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:13:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:13:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:13:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:13:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:13:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:13:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:13:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:13:57.545 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:13:57.547 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.408247_125.1 2023-10-12T19:14:02.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:02.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:02.549 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:02.591 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:02.592 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:02.643 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:02.644 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:02.644 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:02.644 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:02.644 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:02.644 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:02.645 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:02.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:02.645 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:02.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:02.645 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:02.646 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:02.646 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:02.646 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:14:02.646 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:02.646 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:02.647 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:02.647 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:14:02.647 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:02.647 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:02.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:02.648 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:02.648 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:02.648 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:02.648 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:02.648 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:02.649 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:02.649 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:02.649 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:02.649 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:02.649 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:02.650 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:02.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:02.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:02.650 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:02.650 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:02.651 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:02.651 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:02.651 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:02.651 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:02.651 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:02.652 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:02.652 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:02.652 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:02.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:02.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:02.652 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:02.653 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:02.653 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:02.653 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:02.653 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:02.653 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:02.654 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:02.654 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:02.654 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:02.654 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:02.654 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:02.655 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:02.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:02.655 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:02.655 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:02.655 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:02.655 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:02.656 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:02.657 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.435183_128.1 2023-10-12T19:14:07.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:07.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:07.659 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:07.701 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:07.701 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:07.752 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.571704_145.1", 2023-10-12T19:14:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.571704Z", 2023-10-12T19:14:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:07.765 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:07.765 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:07.766 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697137436.143028_223.1 target=1_1697136751.571704_145.1 2023-10-12T19:14:12.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... FAIL 2023-10-12T19:14:12.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:12.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:12.770 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T19:14:12.808 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:12.808 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:12.823 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:12.824 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "ec2c3cfd-ff60-4c7e-8eb0-22a5c29b51a9", 2023-10-12T19:14:12.824 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "remove_zg", 2023-10-12T19:14:12.824 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "remove_zg", 2023-10-12T19:14:12.824 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "false", 2023-10-12T19:14:12.824 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [], 2023-10-12T19:14:12.825 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T19:14:12.825 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T19:14:12.825 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "", 2023-10-12T19:14:12.825 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [], 2023-10-12T19:14:12.825 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [], 2023-10-12T19:14:12.826 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "", 2023-10-12T19:14:12.826 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:14:12.826 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T19:14:12.826 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T19:14:12.826 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:12.827 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:12.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:12.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:12.829 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T19:14:12.906 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:12.907 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:12.932 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:12.932 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "a153ac29-24d6-40e2-a074-03a14f260981", 2023-10-12T19:14:12.932 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "remove", 2023-10-12T19:14:12.933 INFO:teuthology.orchestra.run.smithi170.stdout: "domain_root": "remove.rgw.meta:root", 2023-10-12T19:14:12.933 INFO:teuthology.orchestra.run.smithi170.stdout: "control_pool": "remove.rgw.control", 2023-10-12T19:14:12.933 INFO:teuthology.orchestra.run.smithi170.stdout: "gc_pool": "remove.rgw.log:gc", 2023-10-12T19:14:12.933 INFO:teuthology.orchestra.run.smithi170.stdout: "lc_pool": "remove.rgw.log:lc", 2023-10-12T19:14:12.933 INFO:teuthology.orchestra.run.smithi170.stdout: "log_pool": "remove.rgw.log", 2023-10-12T19:14:12.934 INFO:teuthology.orchestra.run.smithi170.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-10-12T19:14:12.934 INFO:teuthology.orchestra.run.smithi170.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-10-12T19:14:12.934 INFO:teuthology.orchestra.run.smithi170.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-10-12T19:14:12.934 INFO:teuthology.orchestra.run.smithi170.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-10-12T19:14:12.935 INFO:teuthology.orchestra.run.smithi170.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-10-12T19:14:12.935 INFO:teuthology.orchestra.run.smithi170.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-10-12T19:14:12.935 INFO:teuthology.orchestra.run.smithi170.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-10-12T19:14:12.935 INFO:teuthology.orchestra.run.smithi170.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-10-12T19:14:12.935 INFO:teuthology.orchestra.run.smithi170.stdout: "otp_pool": "remove.rgw.otp", 2023-10-12T19:14:12.935 INFO:teuthology.orchestra.run.smithi170.stdout: "system_key": { 2023-10-12T19:14:12.936 INFO:teuthology.orchestra.run.smithi170.stdout: "access_key": "", 2023-10-12T19:14:12.936 INFO:teuthology.orchestra.run.smithi170.stdout: "secret_key": "" 2023-10-12T19:14:12.936 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:12.936 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_pools": [ 2023-10-12T19:14:12.936 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:12.937 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "default-placement", 2023-10-12T19:14:12.937 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:12.937 INFO:teuthology.orchestra.run.smithi170.stdout: "index_pool": "remove.rgw.buckets.index", 2023-10-12T19:14:12.937 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": { 2023-10-12T19:14:12.937 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD": { 2023-10-12T19:14:12.938 INFO:teuthology.orchestra.run.smithi170.stdout: "data_pool": "remove.rgw.buckets.data" 2023-10-12T19:14:12.938 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:12.938 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:12.938 INFO:teuthology.orchestra.run.smithi170.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-10-12T19:14:12.939 INFO:teuthology.orchestra.run.smithi170.stdout: "index_type": 0 2023-10-12T19:14:12.939 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:12.939 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:12.939 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:12.939 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:14:12.940 INFO:teuthology.orchestra.run.smithi170.stdout: "notif_pool": "remove.rgw.log:notif" 2023-10-12T19:14:12.940 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:12.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:12.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:12.942 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T19:14:13.022 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:13.062 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:13.084 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:13.085 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:13.085 INFO:teuthology.orchestra.run.smithi170.stdout: "epoch": 6, 2023-10-12T19:14:13.085 INFO:teuthology.orchestra.run.smithi170.stdout: "predecessor_uuid": "e1332b00-f761-4c62-a5ed-b8b60a8676fd", 2023-10-12T19:14:13.085 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": [], 2023-10-12T19:14:13.085 INFO:teuthology.orchestra.run.smithi170.stdout: "period_map": { 2023-10-12T19:14:13.086 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:13.086 INFO:teuthology.orchestra.run.smithi170.stdout: "zonegroups": [ 2023-10-12T19:14:13.086 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.086 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "a", 2023-10-12T19:14:13.086 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a", 2023-10-12T19:14:13.087 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "", 2023-10-12T19:14:13.087 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "true", 2023-10-12T19:14:13.087 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.087 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T19:14:13.087 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.088 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T19:14:13.088 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T19:14:13.088 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:14:13.088 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [ 2023-10-12T19:14:13.088 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.088 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T19:14:13.089 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a2", 2023-10-12T19:14:13.089 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.089 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8001" 2023-10-12T19:14:13.089 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.090 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:14:13.090 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:14:13.090 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:14:13.090 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:14:13.090 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:14:13.091 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "false", 2023-10-12T19:14:13.091 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:14:13.091 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:14:13.091 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.092 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:14:13.092 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a1", 2023-10-12T19:14:13.092 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.092 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T19:14:13.092 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.092 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:14:13.093 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:14:13.093 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:14:13.093 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:14:13.093 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:14:13.093 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:14:13.094 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:14:13.094 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:14:13.094 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.094 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.094 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [ 2023-10-12T19:14:13.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.095 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default-placement", 2023-10-12T19:14:13.095 INFO:teuthology.orchestra.run.smithi170.stdout: "tags": [], 2023-10-12T19:14:13.095 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": [ 2023-10-12T19:14:13.095 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD" 2023-10-12T19:14:13.095 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:13.096 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.096 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.096 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "default-placement", 2023-10-12T19:14:13.096 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:14:13.096 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T19:14:13.097 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T19:14:13.097 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.097 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "b", 2023-10-12T19:14:13.098 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "b", 2023-10-12T19:14:13.098 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "", 2023-10-12T19:14:13.098 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "false", 2023-10-12T19:14:13.098 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.098 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T19:14:13.099 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.099 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T19:14:13.099 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T19:14:13.099 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T19:14:13.099 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [ 2023-10-12T19:14:13.100 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.100 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T19:14:13.100 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "b2", 2023-10-12T19:14:13.100 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.100 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi187.front.sepia.ceph.com:8001" 2023-10-12T19:14:13.100 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.101 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:14:13.101 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:14:13.101 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:14:13.101 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:14:13.101 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:14:13.102 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:14:13.102 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:14:13.102 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:14:13.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.103 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T19:14:13.103 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "b1", 2023-10-12T19:14:13.103 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.103 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T19:14:13.103 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.104 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:14:13.104 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:14:13.104 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:14:13.104 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:14:13.105 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:14:13.105 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:14:13.106 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:14:13.106 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:14:13.106 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.106 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.106 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [ 2023-10-12T19:14:13.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.107 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default-placement", 2023-10-12T19:14:13.107 INFO:teuthology.orchestra.run.smithi170.stdout: "tags": [], 2023-10-12T19:14:13.107 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": [ 2023-10-12T19:14:13.107 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD" 2023-10-12T19:14:13.108 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:13.108 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.108 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.108 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "default-placement", 2023-10-12T19:14:13.108 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:14:13.109 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T19:14:13.109 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T19:14:13.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.109 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "ec2c3cfd-ff60-4c7e-8eb0-22a5c29b51a9", 2023-10-12T19:14:13.110 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "remove_zg", 2023-10-12T19:14:13.110 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "remove_zg", 2023-10-12T19:14:13.110 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "false", 2023-10-12T19:14:13.110 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [], 2023-10-12T19:14:13.110 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T19:14:13.111 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T19:14:13.111 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "a153ac29-24d6-40e2-a074-03a14f260981", 2023-10-12T19:14:13.111 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [ 2023-10-12T19:14:13.111 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.111 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "a153ac29-24d6-40e2-a074-03a14f260981", 2023-10-12T19:14:13.112 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "remove", 2023-10-12T19:14:13.112 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [], 2023-10-12T19:14:13.112 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:14:13.112 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "false", 2023-10-12T19:14:13.112 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:14:13.113 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:14:13.113 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:14:13.113 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:14:13.113 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:14:13.113 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:14:13.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.114 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.114 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [ 2023-10-12T19:14:13.114 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.114 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default-placement", 2023-10-12T19:14:13.115 INFO:teuthology.orchestra.run.smithi170.stdout: "tags": [], 2023-10-12T19:14:13.115 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": [ 2023-10-12T19:14:13.115 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD" 2023-10-12T19:14:13.115 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:13.115 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.116 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.116 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "default-placement", 2023-10-12T19:14:13.116 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:14:13.116 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T19:14:13.116 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T19:14:13.117 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.117 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.117 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.117 INFO:teuthology.orchestra.run.smithi170.stdout: "short_zone_ids": [ 2023-10-12T19:14:13.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.118 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T19:14:13.118 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 3365504037 2023-10-12T19:14:13.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.118 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "a153ac29-24d6-40e2-a074-03a14f260981", 2023-10-12T19:14:13.119 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 221490571 2023-10-12T19:14:13.119 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.119 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.119 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T19:14:13.119 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 259669302 2023-10-12T19:14:13.120 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.120 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.120 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T19:14:13.120 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 20676319 2023-10-12T19:14:13.120 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.121 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.121 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:14:13.121 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 2601847591 2023-10-12T19:14:13.121 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.121 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:13.122 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.122 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zonegroup": "a", 2023-10-12T19:14:13.122 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:14:13.122 INFO:teuthology.orchestra.run.smithi170.stdout: "period_config": { 2023-10-12T19:14:13.122 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_quota": { 2023-10-12T19:14:13.122 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T19:14:13.123 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T19:14:13.123 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T19:14:13.123 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T19:14:13.123 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T19:14:13.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.124 INFO:teuthology.orchestra.run.smithi170.stdout: "user_quota": { 2023-10-12T19:14:13.124 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T19:14:13.124 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T19:14:13.124 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T19:14:13.124 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T19:14:13.125 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T19:14:13.125 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.125 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:14:13.125 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_name": "test-realm", 2023-10-12T19:14:13.126 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:13.126 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:13.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:13.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:13.127 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T19:14:13.184 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:13.184 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:13.203 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T19:14:13.203+0000 7f69faa1d500 0 zone id a153ac29-24d6-40e2-a074-03a14f260981 is not a part of zonegroup b 2023-10-12T19:14:13.204 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T19:14:13.203+0000 7f69faa1d500 0 zone id a153ac29-24d6-40e2-a074-03a14f260981 is not a part of zonegroup a 2023-10-12T19:14:13.204 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T19:14:13.204+0000 7f69faa1d500 0 zone id a153ac29-24d6-40e2-a074-03a14f260981 is not a part of zonegroup default 2023-10-12T19:14:13.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:13.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:13.211 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T19:14:13.295 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:13.295 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:13.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-12T19:14:13.321+0000 7f879b389500 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-10-12T19:14:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-12T19:14:13.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:13.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:13.328 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T19:14:13.380 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:13.380 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:13.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:13.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T19:14:13.416 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T19:14:13.487 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:13.487 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:13.624 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:13.625 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:13.625 INFO:teuthology.orchestra.run.smithi170.stdout: "epoch": 7, 2023-10-12T19:14:13.625 INFO:teuthology.orchestra.run.smithi170.stdout: "predecessor_uuid": "e1332b00-f761-4c62-a5ed-b8b60a8676fd", 2023-10-12T19:14:13.625 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": [], 2023-10-12T19:14:13.626 INFO:teuthology.orchestra.run.smithi170.stdout: "period_map": { 2023-10-12T19:14:13.626 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:13.626 INFO:teuthology.orchestra.run.smithi170.stdout: "zonegroups": [ 2023-10-12T19:14:13.626 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.626 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "a", 2023-10-12T19:14:13.627 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a", 2023-10-12T19:14:13.627 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "", 2023-10-12T19:14:13.627 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "true", 2023-10-12T19:14:13.627 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.628 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T19:14:13.628 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.628 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T19:14:13.628 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T19:14:13.628 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:14:13.628 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [ 2023-10-12T19:14:13.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.629 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T19:14:13.629 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a2", 2023-10-12T19:14:13.629 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.630 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8001" 2023-10-12T19:14:13.630 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.630 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:14:13.630 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:14:13.631 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:14:13.631 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:14:13.631 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:14:13.631 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "false", 2023-10-12T19:14:13.631 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:14:13.632 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:14:13.632 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.632 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.632 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:14:13.632 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "a1", 2023-10-12T19:14:13.633 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.633 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi170.front.sepia.ceph.com:8000" 2023-10-12T19:14:13.633 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.633 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:14:13.634 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:14:13.634 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:14:13.634 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:14:13.634 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:14:13.634 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:14:13.635 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:14:13.635 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:14:13.635 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.635 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.636 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [ 2023-10-12T19:14:13.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.637 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default-placement", 2023-10-12T19:14:13.637 INFO:teuthology.orchestra.run.smithi170.stdout: "tags": [], 2023-10-12T19:14:13.638 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": [ 2023-10-12T19:14:13.638 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD" 2023-10-12T19:14:13.638 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:13.638 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.638 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.639 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "default-placement", 2023-10-12T19:14:13.639 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:14:13.639 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T19:14:13.639 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T19:14:13.639 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.640 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "b", 2023-10-12T19:14:13.640 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "b", 2023-10-12T19:14:13.640 INFO:teuthology.orchestra.run.smithi170.stdout: "api_name": "", 2023-10-12T19:14:13.641 INFO:teuthology.orchestra.run.smithi170.stdout: "is_master": "false", 2023-10-12T19:14:13.641 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.641 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T19:14:13.641 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.641 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames": [], 2023-10-12T19:14:13.642 INFO:teuthology.orchestra.run.smithi170.stdout: "hostnames_s3website": [], 2023-10-12T19:14:13.642 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T19:14:13.642 INFO:teuthology.orchestra.run.smithi170.stdout: "zones": [ 2023-10-12T19:14:13.642 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.643 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T19:14:13.643 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "b2", 2023-10-12T19:14:13.643 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.643 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi187.front.sepia.ceph.com:8001" 2023-10-12T19:14:13.643 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.644 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:14:13.644 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:14:13.644 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:14:13.644 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:14:13.644 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:14:13.645 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:14:13.645 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:14:13.645 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:14:13.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.645 INFO:teuthology.orchestra.run.smithi170.stdout: "id": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T19:14:13.646 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "b1", 2023-10-12T19:14:13.646 INFO:teuthology.orchestra.run.smithi170.stdout: "endpoints": [ 2023-10-12T19:14:13.646 INFO:teuthology.orchestra.run.smithi170.stdout: "http://smithi187.front.sepia.ceph.com:8000" 2023-10-12T19:14:13.646 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.646 INFO:teuthology.orchestra.run.smithi170.stdout: "log_meta": "false", 2023-10-12T19:14:13.647 INFO:teuthology.orchestra.run.smithi170.stdout: "log_data": "true", 2023-10-12T19:14:13.647 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_index_max_shards": 11, 2023-10-12T19:14:13.647 INFO:teuthology.orchestra.run.smithi170.stdout: "read_only": "false", 2023-10-12T19:14:13.648 INFO:teuthology.orchestra.run.smithi170.stdout: "tier_type": "", 2023-10-12T19:14:13.648 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from_all": "true", 2023-10-12T19:14:13.648 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_from": [], 2023-10-12T19:14:13.649 INFO:teuthology.orchestra.run.smithi170.stdout: "redirect_zone": "" 2023-10-12T19:14:13.649 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.649 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.649 INFO:teuthology.orchestra.run.smithi170.stdout: "placement_targets": [ 2023-10-12T19:14:13.649 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.650 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "default-placement", 2023-10-12T19:14:13.650 INFO:teuthology.orchestra.run.smithi170.stdout: "tags": [], 2023-10-12T19:14:13.650 INFO:teuthology.orchestra.run.smithi170.stdout: "storage_classes": [ 2023-10-12T19:14:13.650 INFO:teuthology.orchestra.run.smithi170.stdout: "STANDARD" 2023-10-12T19:14:13.650 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:13.651 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.651 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.651 INFO:teuthology.orchestra.run.smithi170.stdout: "default_placement": "default-placement", 2023-10-12T19:14:13.651 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:14:13.651 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_policy": { 2023-10-12T19:14:13.652 INFO:teuthology.orchestra.run.smithi170.stdout: "groups": [] 2023-10-12T19:14:13.652 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.652 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.652 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2023-10-12T19:14:13.652 INFO:teuthology.orchestra.run.smithi170.stdout: "short_zone_ids": [ 2023-10-12T19:14:13.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.653 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "56b8a38f-4892-452f-a99f-71629b0b523c", 2023-10-12T19:14:13.653 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 3365504037 2023-10-12T19:14:13.653 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.653 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.653 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "be514f25-5f09-47b0-bc2b-7db6a056f7f7", 2023-10-12T19:14:13.654 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 259669302 2023-10-12T19:14:13.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.654 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.654 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "c344c04e-4982-4ab3-a3f5-b56cbb573cdf", 2023-10-12T19:14:13.654 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 20676319 2023-10-12T19:14:13.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:13.655 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:14:13.655 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 2601847591 2023-10-12T19:14:13.655 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.656 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:13.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.656 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zonegroup": "a", 2023-10-12T19:14:13.656 INFO:teuthology.orchestra.run.smithi170.stdout: "master_zone": "cc69b071-ffc5-4662-8e68-2301691e9014", 2023-10-12T19:14:13.656 INFO:teuthology.orchestra.run.smithi170.stdout: "period_config": { 2023-10-12T19:14:13.656 INFO:teuthology.orchestra.run.smithi170.stdout: "bucket_quota": { 2023-10-12T19:14:13.657 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T19:14:13.657 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T19:14:13.657 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T19:14:13.657 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T19:14:13.657 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T19:14:13.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.658 INFO:teuthology.orchestra.run.smithi170.stdout: "user_quota": { 2023-10-12T19:14:13.658 INFO:teuthology.orchestra.run.smithi170.stdout: "enabled": false, 2023-10-12T19:14:13.658 INFO:teuthology.orchestra.run.smithi170.stdout: "check_on_raw": false, 2023-10-12T19:14:13.658 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size": -1, 2023-10-12T19:14:13.659 INFO:teuthology.orchestra.run.smithi170.stdout: "max_size_kb": 0, 2023-10-12T19:14:13.659 INFO:teuthology.orchestra.run.smithi170.stdout: "max_objects": -1 2023-10-12T19:14:13.659 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:13.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:13.659 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_id": "833c22c1-dbac-4225-ba1b-f4ce971ab379", 2023-10-12T19:14:13.659 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_name": "test-realm", 2023-10-12T19:14:13.660 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:13.660 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:13.660 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-10-12T19:14:13.661 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-36 2023-10-12T19:14:14.679 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-37 2023-10-12T19:14:14.755 INFO:rgw_multi.tests:create bucket zone=b1 name=pqrzfz-38 2023-10-12T19:14:14.792 INFO:rgw_multi.tests:create bucket zone=b2 name=pqrzfz-39 2023-10-12T19:14:15.136 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-40 2023-10-12T19:14:15.150 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-41 2023-10-12T19:14:15.176 INFO:rgw_multi.tests:create bucket zone=b1 name=pqrzfz-42 2023-10-12T19:14:15.215 INFO:rgw_multi.tests:create bucket zone=b2 name=pqrzfz-43 2023-10-12T19:14:18.297 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2023-10-12T19:14:18.298 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-44 2023-10-12T19:14:18.314 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-45 2023-10-12T19:14:18.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:18.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:18.345 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:18.423 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T19:14:18.435 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T19:14:18.435 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:18.435 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697138058.331751_285.1", 2023-10-12T19:14:18.435 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T19:14:18.331751Z" 2023-10-12T19:14:18.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:18.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:18.436 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:18.436 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T19:14:18.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:18.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:18.437 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:18.437 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:14:18.437 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:18.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:18.438 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:18.438 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:14:18.438 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:18.438 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T19:14:18.439 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T19:14:18.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:18.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:18.441 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:18.521 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:18.521 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:18.572 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:18.572 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:18.573 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:18.573 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:18.573 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:18.573 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:18.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:18.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:18.574 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:18.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:18.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:18.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:18.575 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:18.575 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:14:18.575 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:18.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:18.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:18.576 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:14:18.576 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:18.576 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:18.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:18.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:18.577 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:18.577 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:18.577 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:18.577 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:18.578 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:18.578 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:18.578 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:18.578 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:18.579 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:18.579 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:18.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:18.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:18.579 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:18.579 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:18.580 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:18.580 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:18.580 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:18.580 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:18.580 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:18.581 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:18.581 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:18.581 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:18.581 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:18.581 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:18.582 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:18.582 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:18.582 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:18.582 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:18.582 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:18.583 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:18.583 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:18.583 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:18.583 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:18.583 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:18.584 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:18.584 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:18.584 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:18.584 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:18.584 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:18.585 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:18.585 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:18.585 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:18.586 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:14:23.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:23.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:23.588 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:23.637 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:23.637 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:23.695 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:23.695 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:23.696 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:23.696 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:23.696 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:23.696 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:23.696 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:23.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:23.697 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:23.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:23.697 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:23.698 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:23.698 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:23.698 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:14:23.698 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:23.698 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:23.699 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:23.699 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:14:23.699 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:23.699 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:23.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:23.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:23.700 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:23.700 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:23.700 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:23.701 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:23.701 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:23.701 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:23.701 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:23.702 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:23.702 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:23.702 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:23.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:23.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:23.703 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:23.703 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:23.703 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:23.703 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:23.703 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:23.704 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:23.704 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:23.704 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:23.704 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:23.704 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:23.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:23.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:23.705 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:23.705 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:23.705 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:23.705 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:23.706 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:23.706 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:23.706 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:23.706 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:23.706 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:23.707 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:23.707 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:23.707 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:23.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:23.707 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:23.708 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:23.708 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:23.708 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:23.708 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:23.709 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:14:28.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:28.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:28.712 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:28.760 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:28.760 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:28.818 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:28.818 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:28.818 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:28.819 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:28.820 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:28.820 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:28.820 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:28.820 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:28.820 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:28.821 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:14:28.821 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:28.821 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:28.821 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:28.821 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:14:28.822 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:28.822 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:28.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:28.822 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:28.822 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:28.823 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:28.823 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:28.823 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:28.823 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:28.823 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:28.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:28.825 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:28.826 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:28.827 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:28.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:28.827 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:28.827 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:28.827 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:28.828 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:28.828 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:28.828 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:28.828 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:28.829 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:28.829 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:28.829 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:28.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:28.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:28.830 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:28.830 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:28.830 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:28.830 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:28.830 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:28.831 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:28.831 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:28.832 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:14:33.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:33.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:33.835 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:33.885 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:33.885 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:33.942 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:33.942 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:33.943 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:33.943 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:33.943 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:33.943 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:33.943 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:33.944 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:33.944 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:33.944 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:33.944 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:33.944 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:33.945 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:33.945 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:14:33.945 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:33.945 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:33.945 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:33.946 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:14:33.946 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:33.946 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:33.946 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:33.946 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:33.947 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:33.947 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:33.947 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:33.947 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:33.947 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:33.948 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:33.948 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:33.948 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:33.948 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:33.948 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:33.949 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:33.949 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:33.949 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:33.949 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:33.950 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:33.950 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:33.950 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:33.950 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:33.951 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:33.951 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:33.951 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:33.951 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:33.951 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:33.952 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:33.953 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:33.953 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:33.953 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:33.953 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:33.953 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:33.954 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:33.954 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:33.954 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:33.954 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:33.954 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:33.955 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:33.955 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:33.955 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:33.955 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:33.956 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:14:38.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:38.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:38.959 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:39.007 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:39.007 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:39.070 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:39.070 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:39.071 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:39.071 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:39.071 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:39.071 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:39.071 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:39.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:39.072 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:39.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:39.072 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:39.072 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:39.072 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:39.073 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:14:39.073 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:39.073 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:39.073 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:39.073 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:14:39.074 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:39.074 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:39.074 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:39.074 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:39.075 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:39.075 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:39.075 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:39.075 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:39.076 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:39.076 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:39.076 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:39.076 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:39.076 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:39.077 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:39.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:39.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:39.077 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:39.077 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:39.077 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:39.078 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:39.078 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:39.078 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:39.078 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:39.078 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:39.079 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:39.079 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:39.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:39.079 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:39.079 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:39.080 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:39.080 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:39.080 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:39.080 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:39.080 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:39.081 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:39.081 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:39.081 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:39.081 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:39.081 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:39.081 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:39.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:39.082 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:39.082 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:39.082 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:39.082 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:39.083 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:39.084 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:14:44.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:44.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:44.086 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:44.134 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:44.135 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:44.192 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:44.192 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:44.192 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:44.193 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:44.193 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:44.193 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:44.193 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:44.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:44.194 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:44.194 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:44.194 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:44.194 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:44.194 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:44.195 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:14:44.195 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:44.195 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:44.195 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:44.196 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:14:44.196 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:44.196 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:44.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:44.197 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:44.197 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:44.197 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:44.198 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:44.198 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:44.198 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:44.198 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:44.199 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:44.199 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:44.199 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:44.199 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:44.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:44.200 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:44.200 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:44.200 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:44.200 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:44.200 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:44.201 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:44.201 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:44.201 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:44.201 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:44.201 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:44.202 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:44.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:44.202 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:44.202 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:44.202 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:44.202 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:44.203 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:44.203 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:44.203 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:44.203 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:44.203 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:44.204 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:44.204 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:44.204 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:44.204 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:44.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:44.205 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:44.205 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:44.205 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:44.205 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:44.205 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:44.207 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:14:49.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:49.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:49.209 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:49.256 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:49.256 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:49.313 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:49.313 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:49.313 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:49.313 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:49.314 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:49.314 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:49.314 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:49.314 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:49.315 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:49.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:49.315 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:49.315 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:49.315 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:49.316 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:14:49.316 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:49.316 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:49.316 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:49.316 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:14:49.317 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:49.317 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:49.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:49.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:49.317 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:49.318 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:49.318 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:49.318 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:49.318 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:49.318 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:49.319 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:49.319 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:49.319 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:49.319 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:49.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:49.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:49.320 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:49.320 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:49.320 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:49.320 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:49.321 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:49.321 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:49.321 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:49.321 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:49.321 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:49.322 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:49.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:49.322 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:49.322 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:49.323 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:49.323 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:49.323 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:49.323 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:49.323 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:49.324 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:49.324 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:49.324 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:49.324 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:49.324 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:49.325 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:49.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:49.325 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:49.325 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:49.325 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:49.325 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:49.326 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:49.327 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:14:54.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:54.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:54.329 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:54.373 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:54.373 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:54.424 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:54.424 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:54.425 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:54.425 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:54.425 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:54.425 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:54.426 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:54.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:54.426 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:54.426 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:54.426 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:54.427 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:54.427 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:54.427 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:14:54.427 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:54.427 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:54.428 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:54.428 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:14:54.428 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:54.428 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:54.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:54.429 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:54.429 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:54.429 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:54.429 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:54.429 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:54.429 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:54.430 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:54.430 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:54.430 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:54.430 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:54.430 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:54.431 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:54.431 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:54.431 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:54.431 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:54.432 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:54.432 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:54.432 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:54.432 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:54.432 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:54.432 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:54.433 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:54.433 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:54.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:54.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:54.433 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:54.434 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:54.434 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:54.434 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:54.434 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:54.434 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:54.435 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:54.435 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:54.435 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:54.435 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:54.435 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:54.436 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:54.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:54.436 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:54.436 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:54.436 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:54.436 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:54.437 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:54.438 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:14:59.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:59.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:14:59.440 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:14:59.488 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:14:59.489 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:14:59.545 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:14:59.545 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:14:59.545 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:14:59.546 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:14:59.546 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:14:59.546 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:14:59.546 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:59.547 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:59.547 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:14:59.547 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:59.547 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:14:59.547 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:59.548 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:59.548 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:14:59.548 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:59.548 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:14:59.548 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:59.549 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:14:59.549 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:59.549 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:59.549 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:59.549 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:59.550 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:14:59.550 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:59.550 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:59.550 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:14:59.550 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:59.551 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:59.551 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:59.551 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:14:59.551 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:59.552 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:59.552 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:59.553 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:59.553 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:14:59.553 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:59.554 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:59.554 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:59.554 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:59.554 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:59.555 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:59.555 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:59.555 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:59.555 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:59.555 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:59.555 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:14:59.556 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:14:59.556 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:14:59.556 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:14:59.556 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:14:59.557 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:14:59.557 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:14:59.557 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:14:59.557 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:14:59.557 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:14:59.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:59.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:59.558 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:14:59.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:14:59.558 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:14:59.559 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:14:59.559 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:14:59.559 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:14:59.559 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:14:59.560 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:04.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:04.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:04.566 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:04.614 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:04.614 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:04.670 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:04.671 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:04.671 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:04.671 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:04.671 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:04.671 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:04.672 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:04.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:04.672 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:04.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:04.673 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:04.673 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:04.673 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:04.673 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:04.673 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:04.674 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:04.674 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:04.674 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:04.674 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:04.674 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:04.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:04.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:04.675 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:04.675 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:04.675 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:04.676 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:04.676 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:04.676 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:04.676 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:04.676 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:04.676 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:04.677 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:04.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:04.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:04.677 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:04.677 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:04.678 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:04.678 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:04.678 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:04.678 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:04.678 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:04.679 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:04.679 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:04.679 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:04.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:04.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:04.680 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:04.680 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:04.680 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:04.680 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:04.680 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:04.680 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:04.681 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:04.681 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:04.681 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:04.681 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:04.681 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:04.682 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:04.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:04.682 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:04.682 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:04.682 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:04.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:04.683 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:04.684 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:09.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:09.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:09.686 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:09.736 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:09.736 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:09.794 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:09.795 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:09.795 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:09.795 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:09.795 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:09.796 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:09.796 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:09.796 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:09.796 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:09.796 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:09.797 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:09.797 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:09.797 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:09.797 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:09.797 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:09.798 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:09.798 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:09.798 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:09.798 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:09.798 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:09.799 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:09.799 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:09.799 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:09.799 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:09.800 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:09.800 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:09.800 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:09.800 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:09.801 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:09.801 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:09.801 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:09.801 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:09.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:09.802 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:09.802 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:09.802 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:09.802 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:09.803 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:09.803 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:09.803 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:09.803 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:09.803 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:09.804 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:09.804 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:09.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:09.804 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:09.804 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:09.805 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:09.805 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:09.805 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:09.805 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:09.805 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:09.805 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:09.806 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:09.806 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:09.806 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:09.806 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:09.806 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:09.807 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:09.807 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:09.807 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:09.807 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:09.807 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:09.808 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:09.809 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:14.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:14.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:14.811 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:14.859 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:14.859 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:14.915 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:14.916 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:14.916 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:14.916 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:14.916 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:14.918 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:14.918 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:14.918 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:14.918 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:14.918 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:14.923 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:14.923 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:14.923 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:14.923 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:14.923 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:14.924 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:14.924 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:14.924 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:14.924 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:14.924 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:14.924 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:14.925 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:14.925 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:14.925 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:14.925 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:14.925 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:14.926 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:14.926 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:14.926 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:14.926 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:14.926 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:14.927 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:14.927 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:14.927 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:14.927 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:14.927 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:14.928 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:14.928 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:14.929 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:19.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:19.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:19.931 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:19.979 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:19.979 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:20.037 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:20.037 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:20.037 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:20.037 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:20.038 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:20.038 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:20.038 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:20.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:20.038 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:20.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:20.039 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:20.039 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:20.039 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:20.039 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:20.040 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:20.040 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:20.040 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:20.040 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:20.040 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:20.041 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:20.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:20.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:20.041 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:20.041 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:20.042 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:20.042 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:20.042 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:20.042 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:20.042 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:20.042 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:20.043 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:20.043 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:20.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:20.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:20.043 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:20.044 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:20.044 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:20.044 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:20.044 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:20.044 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:20.045 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:20.045 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:20.045 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:20.045 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:20.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:20.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:20.046 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:20.046 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:20.046 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:20.046 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:20.046 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:20.047 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:20.047 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:20.047 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:20.047 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:20.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:20.048 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:20.048 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:20.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:20.048 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:20.048 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:20.049 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:20.049 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:20.049 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:20.050 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:25.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:25.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:25.052 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:25.096 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:25.096 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:25.148 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:25.148 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:25.149 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:25.149 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:25.149 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:25.149 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:25.150 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:25.150 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:25.150 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:25.150 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:25.150 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:25.151 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:25.151 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:25.151 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:25.151 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:25.151 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:25.152 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:25.152 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:25.152 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:25.152 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:25.152 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:25.153 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:25.153 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:25.153 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:25.153 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:25.153 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:25.154 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:25.154 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:25.154 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:25.154 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:25.154 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:25.155 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:25.155 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:25.155 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:25.155 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:25.156 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:25.156 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:25.156 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:25.156 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:25.156 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:25.157 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:25.157 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:25.157 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:25.157 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:25.157 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:25.158 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:25.158 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:25.158 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:25.158 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:25.159 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:25.159 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:25.159 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:25.159 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:25.159 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:25.159 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:25.160 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:25.160 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:25.160 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:25.160 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:25.160 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:25.161 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:25.161 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:25.161 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:25.161 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:25.162 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:30.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:30.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:30.165 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:30.213 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:30.213 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:30.274 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:30.275 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:30.275 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:30.275 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:30.275 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:30.275 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:30.276 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:30.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:30.276 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:30.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:30.276 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:30.277 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:30.277 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:30.277 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:30.277 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:30.277 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:30.278 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:30.278 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:30.278 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:30.278 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:30.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:30.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:30.279 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:30.279 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:30.279 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:30.280 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:30.280 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:30.280 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:30.280 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:30.280 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:30.281 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:30.281 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:30.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:30.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:30.281 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:30.282 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:30.282 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:30.282 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:30.282 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:30.282 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:30.283 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:30.283 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:30.283 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:30.283 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:30.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:30.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:30.284 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:30.284 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:30.284 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:30.284 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:30.285 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:30.285 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:30.285 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:30.285 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:30.285 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:30.286 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:30.286 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:30.286 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:30.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:30.286 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:30.287 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:30.287 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:30.287 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:30.287 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:30.288 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:35.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:35.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:35.291 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:35.339 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:35.340 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:35.396 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:35.396 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:35.397 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:35.397 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:35.397 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:35.397 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:35.397 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:35.398 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:35.398 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:35.398 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:35.398 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:35.398 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:35.399 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:35.399 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:35.399 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:35.399 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:35.399 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:35.400 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:35.400 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:35.400 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:35.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:35.400 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:35.400 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:35.401 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:35.401 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:35.401 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:35.401 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:35.401 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:35.402 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:35.402 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:35.402 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:35.402 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:35.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:35.403 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:35.403 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:35.403 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:35.403 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:35.403 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:35.404 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:35.404 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:35.404 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:35.404 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:35.404 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:35.404 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:35.405 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:35.405 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:35.405 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:35.405 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:35.405 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:35.406 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:35.406 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:35.406 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:35.406 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:35.406 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:35.407 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:35.407 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:35.407 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:35.407 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:35.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:35.407 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:35.408 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:35.408 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:35.408 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:35.408 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:35.409 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:40.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:40.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:40.412 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:40.457 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:40.457 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:40.510 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:40.511 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:40.511 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:40.511 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:40.513 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:40.513 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:40.513 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:40.513 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:40.514 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:40.514 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:40.514 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:40.514 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:40.514 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:40.515 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:40.515 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:40.515 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:40.515 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:40.515 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:40.516 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:40.516 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:40.516 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:40.516 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:40.516 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:40.517 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:40.517 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:40.517 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:40.517 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:40.517 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:40.518 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:40.518 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:40.518 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:40.518 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:40.518 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:40.519 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:40.519 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:40.519 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:40.519 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:40.519 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:40.520 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:40.520 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:40.520 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:40.520 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:40.520 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:40.521 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:40.521 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:40.521 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:40.521 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:40.521 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:40.522 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:40.522 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:40.522 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:40.522 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:40.522 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:40.523 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:40.523 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:40.523 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:40.523 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:40.523 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:40.524 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:45.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:45.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:45.527 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:45.575 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:45.575 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:45.633 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:45.633 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:45.633 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:45.634 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:45.634 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:45.634 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:45.634 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:45.634 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:45.635 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:45.635 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:45.635 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:45.635 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:45.635 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:45.636 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:45.636 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:45.636 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:45.636 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:45.636 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:45.637 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:45.637 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:45.637 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:45.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:45.637 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:45.638 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:45.638 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:45.638 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:45.638 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:45.638 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:45.639 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:45.639 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:45.639 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:45.639 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:45.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:45.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:45.640 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:45.640 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:45.641 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:45.641 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:45.641 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:45.641 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:45.641 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:45.642 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:45.642 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:45.642 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:45.642 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:45.642 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:45.643 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:45.643 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:45.643 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:45.643 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:45.643 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:45.643 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:45.644 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:45.644 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:45.644 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:45.644 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:45.644 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:45.644 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:45.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:45.645 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:45.645 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:45.645 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:45.645 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:45.646 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:45.647 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:50.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:50.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:50.649 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:50.694 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:50.695 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:50.748 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:50.749 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:50.749 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:50.749 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:50.749 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:50.749 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:50.750 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:50.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:50.750 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:50.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:50.750 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:50.751 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:50.751 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:50.751 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:50.751 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:50.751 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:50.752 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:50.752 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:50.752 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:50.752 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:50.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:50.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:50.753 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:50.753 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:50.753 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:50.753 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:50.753 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:50.754 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:50.754 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:50.754 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:50.754 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:50.755 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:50.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:50.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:50.755 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:50.755 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:50.755 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:50.756 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:50.756 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:50.756 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:50.756 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:50.756 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:50.757 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:50.757 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:50.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:50.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:50.757 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:50.757 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:50.758 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:50.758 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:50.758 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:50.758 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:50.759 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:50.759 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:50.759 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:50.759 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:50.760 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:50.760 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:50.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:50.760 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:50.760 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:50.761 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:50.761 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:50.761 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:50.762 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:15:55.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:55.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:15:55.765 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:15:55.811 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:15:55.812 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:15:55.867 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:15:55.867 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:15:55.868 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:15:55.868 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:15:55.868 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:15:55.868 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:15:55.869 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:55.869 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:55.869 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:15:55.869 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:55.869 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:15:55.870 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:55.870 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:55.870 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:15:55.870 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:55.870 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:15:55.871 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:55.871 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:15:55.871 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:55.871 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:55.871 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:55.872 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:55.872 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:15:55.872 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:55.872 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:55.872 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:15:55.872 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:55.873 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:55.873 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:55.873 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:15:55.873 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:55.874 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:55.874 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:55.874 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:55.874 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:15:55.875 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:55.875 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:55.875 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:55.875 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:55.876 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:55.876 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:55.876 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:55.876 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:55.876 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:55.876 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:55.877 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:15:55.877 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:15:55.877 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:15:55.877 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:15:55.877 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:15:55.878 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:15:55.878 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:15:55.878 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:15:55.878 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:15:55.878 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:15:55.879 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:55.879 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:55.879 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:15:55.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:15:55.879 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:15:55.880 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:15:55.880 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:15:55.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:15:55.880 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:15:55.881 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:16:00.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:00.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:00.884 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:00.932 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:00.932 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:00.989 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:00.990 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:00.991 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:00.991 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:00.991 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:00.991 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:00.991 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:00.992 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:00.992 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:00.992 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:16:00.992 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:00.992 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:00.993 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:00.993 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:16:00.993 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:00.993 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:00.993 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:00.994 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:00.994 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:00.994 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:00.994 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:00.995 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:00.995 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:00.995 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:00.995 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:00.995 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:00.995 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:00.996 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:00.996 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:00.996 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:00.996 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:00.996 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:00.997 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:00.997 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:00.997 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:00.997 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:00.997 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:00.998 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:00.998 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:00.998 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:00.998 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:00.999 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:00.999 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:00.999 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:00.999 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:00.999 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:01.000 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:01.000 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:01.000 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:01.000 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:01.000 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:01.000 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:01.001 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:01.001 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:01.001 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:01.001 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:01.001 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:01.002 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:01.002 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:01.002 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:01.003 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:16:06.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:06.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:06.006 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:06.053 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:06.054 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:06.111 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:06.111 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:06.111 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:06.112 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:06.112 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:06.112 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:06.112 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:06.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:06.113 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:06.113 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:06.113 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:06.113 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:06.113 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:06.114 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:16:06.114 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:06.114 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:06.114 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:06.114 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:16:06.115 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:06.115 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:06.115 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:06.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:06.115 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:06.116 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:06.116 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:06.116 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:06.116 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:06.116 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:06.117 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:06.117 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:06.117 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:06.117 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:06.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:06.119 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:06.119 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:06.119 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:06.119 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:06.119 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:06.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:06.123 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:06.123 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:06.123 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:06.123 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:06.123 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:06.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:16:11.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:11.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:11.127 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:11.173 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:11.173 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:11.228 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:16:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:16:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:11.240 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:11.240 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:11.240 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:11.240 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:11.241 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:16:16.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:16.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:16.244 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:16.291 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:16.292 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:16.346 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:16.347 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:16.347 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:16.347 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:16.348 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:16.348 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:16.348 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:16.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:16.349 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:16.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:16.349 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:16.349 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:16.350 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:16.350 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.404891_123.1", 2023-10-12T19:16:16.350 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:16.350 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:16.350 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:16.351 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.404891Z", 2023-10-12T19:16:16.351 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:16.351 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:16.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:16.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:16.352 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:16.352 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:16.352 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:16.352 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:16.352 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:16.353 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:16.353 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:16.353 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:16.353 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:16.354 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:16.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:16.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:16.354 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:16.355 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:16.355 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:16.355 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:16.355 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:16.356 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:16.356 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:16.356 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:16.356 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:16.356 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:16.357 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:16.357 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:16.357 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:16.357 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:16.357 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:16.358 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:16.359 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:16.363 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:16.363 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:16.363 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:16.363 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:16.364 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:16.364 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:16.364 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:16.364 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:16.365 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:16.366 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.404891_123.1 2023-10-12T19:16:21.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:21.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:21.368 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:21.438 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:21.439 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:21.799 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:21.799 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:21.799 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:21.800 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:21.800 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:21.800 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:21.800 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:21.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:21.801 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:21.801 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:21.801 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:21.801 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:21.801 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:21.802 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.492727_137.1", 2023-10-12T19:16:21.802 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:21.802 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:21.802 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:21.802 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.492727Z", 2023-10-12T19:16:21.803 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:21.803 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:21.803 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:21.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:21.803 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:21.804 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:21.804 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:21.804 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:21.804 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:21.804 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:21.805 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:21.805 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:21.805 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:21.805 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:21.805 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:21.806 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:21.806 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:21.806 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:21.806 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:21.806 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:21.807 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:21.807 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:21.807 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:21.807 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:21.807 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:21.808 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:21.808 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:21.808 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:21.808 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:21.808 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:21.809 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:21.809 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:21.809 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:21.809 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:21.809 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:21.810 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:21.810 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:21.810 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:21.810 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:21.811 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:21.811 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:21.811 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:21.811 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:21.811 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:21.812 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:21.812 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:21.812 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.492727_137.1 2023-10-12T19:16:26.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:26.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:26.815 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:26.898 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:26.898 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:27.119 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:27.119 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:27.119 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:27.120 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:27.120 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:27.120 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:27.120 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:27.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:27.121 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:27.121 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:27.121 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:27.121 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:27.122 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:27.122 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T19:16:27.123 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:27.123 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:27.123 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:27.124 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T19:16:27.124 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:27.124 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:27.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:27.125 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:27.125 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:27.125 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:27.126 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:27.126 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:27.126 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:27.127 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:27.127 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:27.127 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:27.127 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:27.128 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:27.128 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:27.128 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:27.129 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:27.129 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:27.129 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:27.129 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:27.129 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:27.130 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:27.130 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:27.130 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:27.130 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:27.130 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:27.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:27.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:27.131 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:27.131 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:27.131 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:27.131 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:27.132 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:27.132 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:27.132 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:27.132 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:27.132 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:27.133 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:27.133 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:27.133 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:27.133 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:27.133 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:27.134 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:27.134 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:27.134 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:27.134 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:27.136 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.438188_130.1 2023-10-12T19:16:32.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:32.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:32.140 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:32.246 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:32.246 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:32.524 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:32.524 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:32.524 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:32.524 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:32.525 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:32.525 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:32.525 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:32.525 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:32.525 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:32.526 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:32.526 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:32.526 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:32.526 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:32.527 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.488676_136.1", 2023-10-12T19:16:32.527 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:32.527 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:32.527 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:32.527 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.488676Z", 2023-10-12T19:16:32.528 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:32.528 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:32.528 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:32.528 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:32.528 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:32.529 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:32.529 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:32.529 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:32.529 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:32.529 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:32.530 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:32.530 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:32.530 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:32.530 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:32.530 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:32.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:32.531 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:32.531 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:32.531 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:32.531 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:32.532 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:32.532 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:32.532 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:32.532 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:32.532 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:32.532 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:32.533 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:32.533 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:32.533 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:32.533 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:32.533 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:32.534 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:32.534 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:32.534 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:32.534 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:32.534 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:32.535 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:32.535 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:32.535 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:32.535 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:32.536 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:32.536 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:32.536 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:32.536 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:32.536 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:32.537 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:32.538 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.488676_136.1 2023-10-12T19:16:37.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:37.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:37.540 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:37.583 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:37.583 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:37.653 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:37.654 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:37.654 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:37.654 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:37.654 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:37.654 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:37.655 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:37.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:37.655 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:37.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:37.655 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:37.656 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:37.656 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:37.656 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.547705_140.1", 2023-10-12T19:16:37.656 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:37.656 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:37.657 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:37.657 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.547705Z", 2023-10-12T19:16:37.657 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:37.657 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:37.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:37.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:37.658 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:37.658 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:37.658 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:37.658 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:37.658 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:37.659 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:37.659 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:37.659 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:37.659 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:37.660 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:37.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:37.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:37.660 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:37.660 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:37.661 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:37.661 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:37.661 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:37.661 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:37.661 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:37.662 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:37.662 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:37.662 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:37.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:37.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:37.663 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:37.663 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:37.663 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:37.663 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:37.663 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:37.663 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:37.664 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:37.664 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:37.664 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:37.664 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:37.664 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:37.665 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:37.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:37.665 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:37.665 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:37.665 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:37.665 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:37.666 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:37.669 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.547705_140.1 2023-10-12T19:16:42.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:42.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:42.672 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:42.714 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:42.714 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:42.766 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:42.766 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:42.767 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:42.767 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:42.767 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:42.767 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:42.767 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:42.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:42.768 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:42.768 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:42.768 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:42.768 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:42.769 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:42.769 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:16:42.769 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:42.769 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:42.769 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:42.770 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:16:42.770 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:42.770 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:42.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:42.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:42.771 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:42.771 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:42.771 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:42.771 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:42.771 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:42.772 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:42.772 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:42.772 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:42.772 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:42.772 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:42.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:42.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:42.773 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:42.773 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:42.773 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:42.774 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:42.774 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:42.774 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:42.774 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:42.775 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:42.775 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:42.775 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:42.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:42.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:42.776 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:42.776 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:42.776 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:42.776 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:42.777 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:42.777 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:42.777 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:42.777 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:42.777 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:42.778 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:42.778 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:42.778 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:42.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:42.778 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:42.779 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:42.779 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:42.779 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:42.779 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:42.782 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.470944_133.1 2023-10-12T19:16:47.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:47.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:47.785 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:47.827 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:47.827 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:47.879 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:47.879 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:47.880 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:47.880 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:47.880 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:47.880 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:47.881 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:47.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:47.881 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:47.881 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:47.881 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:47.882 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:47.882 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:47.882 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:16:47.882 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:47.883 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:47.883 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:47.883 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:16:47.883 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:47.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:47.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:47.884 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:47.884 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:47.884 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:47.884 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:47.884 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:47.885 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:47.885 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:47.885 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:47.885 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:47.885 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:47.886 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:47.886 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:47.886 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:47.886 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:47.886 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:47.887 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:47.887 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:47.887 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:47.887 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:47.887 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:47.888 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:47.888 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:47.888 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:47.888 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:47.888 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:47.889 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:47.889 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:47.889 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:47.889 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:47.889 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:47.890 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:47.890 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:47.890 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:47.890 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:47.890 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:47.891 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:47.891 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:47.891 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:47.891 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:47.891 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:47.892 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:47.892 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:47.892 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:47.893 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:16:52.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:52.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:52.896 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:52.984 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:52.985 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:53.290 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:53.290 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:53.290 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:53.290 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:53.291 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:53.291 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:53.291 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:53.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:53.291 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:53.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:53.292 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:53.292 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:53.292 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:53.293 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.486122_135.1", 2023-10-12T19:16:53.293 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:53.293 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:53.293 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:53.293 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.486122Z", 2023-10-12T19:16:53.294 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:53.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:53.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:53.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:53.295 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:53.295 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:53.295 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:53.295 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:53.295 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:53.295 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:53.296 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:53.296 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:53.296 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:53.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:53.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:53.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:53.297 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:53.297 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:53.297 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:53.297 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:53.298 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:53.298 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:53.298 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:53.298 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:53.298 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:53.299 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:53.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:53.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:53.299 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:53.300 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:53.300 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:53.300 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:53.300 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:53.300 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:53.300 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:53.301 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:53.301 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:53.301 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:53.301 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:53.301 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:53.302 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:53.302 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:53.302 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:53.302 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:53.303 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:53.303 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:53.304 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.486122_135.1 2023-10-12T19:16:58.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:58.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:16:58.306 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:16:58.354 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:16:58.354 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:16:58.442 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:16:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:16:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:16:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:16:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:16:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:16:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:16:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:16:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:16:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:16:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:16:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:16:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:16:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:16:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:16:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:16:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:16:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:16:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:16:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:16:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:16:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:16:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:16:58.454 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:16:58.454 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:16:58.454 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:58.454 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:58.454 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:16:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:16:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:16:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:16:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:16:58.456 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:16:58.456 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:16:58.457 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:17:03.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:03.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:03.459 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:03.544 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:03.544 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:03.985 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:03.986 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:03.986 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:03.986 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:03.986 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:03.987 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:03.987 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:03.987 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:03.987 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:03.987 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:03.988 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:03.988 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:03.988 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:03.988 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:17:03.989 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:03.989 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:03.989 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:03.989 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:17:03.989 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:03.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:03.990 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:03.990 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:03.990 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:03.991 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:03.991 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:03.991 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:03.991 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:03.992 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:03.992 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:03.992 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:03.992 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:03.993 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:03.993 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:03.993 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:03.993 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:03.994 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:03.994 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:03.994 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:03.994 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:03.995 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:03.995 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:03.995 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:03.995 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:03.995 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:03.995 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:03.996 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:03.996 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:03.996 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:03.997 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:03.997 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:03.997 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:03.997 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:03.998 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:03.998 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:03.998 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:03.998 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:03.998 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:03.999 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:03.999 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:03.999 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:03.999 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:03.999 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:03.999 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:04.000 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:04.001 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.407338_124.1 2023-10-12T19:17:09.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:09.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:09.004 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:09.116 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:09.116 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:09.429 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:09.430 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:09.430 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:09.430 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:09.430 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:09.431 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:09.431 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:09.431 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:09.431 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:09.431 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:09.432 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:09.432 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:09.432 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:09.432 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:17:09.433 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:09.433 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:09.433 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:09.433 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:17:09.433 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:09.434 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:09.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:09.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:09.434 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:09.434 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:09.435 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:09.435 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:09.435 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:09.435 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:09.436 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:09.436 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:09.436 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:09.436 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:09.437 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:09.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:09.437 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:09.437 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:09.438 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:09.438 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:09.438 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:09.438 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:09.438 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:09.439 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:09.439 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:09.439 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:09.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:09.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:09.440 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:09.440 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:09.440 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:09.441 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:09.441 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:09.441 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:09.441 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:09.442 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:09.442 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:09.442 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:09.443 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:09.443 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:09.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:09.443 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:09.444 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:09.444 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:09.444 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:09.444 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:09.446 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:17:14.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:14.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:14.449 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:14.493 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:14.493 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:14.911 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:14.911 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:14.911 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:14.911 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:14.912 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:14.912 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:14.912 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:14.912 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:14.912 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:14.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:14.913 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:14.913 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:14.913 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:14.913 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.547705_140.1", 2023-10-12T19:17:14.914 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:14.914 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:14.914 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:14.914 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.547705Z", 2023-10-12T19:17:14.914 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:14.915 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:14.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:14.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:14.915 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:14.915 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:14.916 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:14.916 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:14.916 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:14.916 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:14.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:14.917 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:14.918 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:14.918 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:14.918 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:14.919 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:14.920 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:14.921 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:14.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:14.923 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:14.923 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:14.923 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:14.923 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:14.923 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:14.924 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.547705_140.1 2023-10-12T19:17:19.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:19.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:19.927 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:20.011 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:20.011 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:20.403 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:20.403 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:20.403 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:20.404 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:20.404 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:20.404 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:20.404 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:20.404 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:20.405 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:20.405 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:20.405 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:20.405 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:20.405 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:20.406 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:17:20.406 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:20.406 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:20.406 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:20.406 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:17:20.407 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:20.407 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:20.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:20.407 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:20.407 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:20.408 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:20.408 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:20.408 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:20.408 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:20.408 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:20.409 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:20.409 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:20.409 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:20.409 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:20.409 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:20.410 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:20.410 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:20.410 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:20.410 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:20.410 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:20.411 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:20.411 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:20.411 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:20.411 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:20.411 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:20.412 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:20.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:20.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:20.412 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:20.412 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:20.413 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:20.413 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:20.413 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:20.413 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:20.413 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:20.414 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:20.414 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:20.414 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:20.414 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:20.414 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:20.414 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:20.415 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:20.415 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:20.415 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:20.415 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:20.415 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:20.416 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.410419_126.1 2023-10-12T19:17:25.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:25.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:25.420 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:25.464 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:25.464 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:25.645 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:25.646 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:25.646 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:25.646 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:25.647 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:25.647 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:25.647 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:25.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:25.648 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:25.648 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:25.648 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:25.648 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:25.649 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:25.649 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:17:25.649 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:25.649 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:25.650 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:25.650 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:17:25.650 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:25.650 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:25.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:25.651 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:25.651 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:25.651 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:25.651 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:25.651 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:25.652 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:25.652 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:25.652 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:25.652 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:25.652 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:25.653 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:25.653 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:25.653 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:25.653 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:25.653 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:25.654 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:25.654 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:25.654 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:25.654 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:25.655 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:25.655 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:25.655 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:25.656 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:25.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:25.656 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:25.656 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:25.656 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:25.657 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:25.657 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:25.657 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:25.657 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:25.657 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:25.657 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:25.658 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:25.658 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:25.658 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:25.658 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:25.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:25.659 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:25.659 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:25.659 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:25.659 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:25.660 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:25.661 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.470944_133.1 2023-10-12T19:17:30.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:30.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:30.664 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:30.764 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:30.765 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:30.765 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:30.765 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:30.765 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:30.765 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:30.766 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:30.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:30.766 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:30.766 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:30.766 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:30.767 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:30.767 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:30.767 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:17:30.767 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:30.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:30.769 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:30.769 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:30.769 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:30.769 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:30.770 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:30.770 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:30.770 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:30.770 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:30.770 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:30.771 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:30.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:30.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:30.771 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:30.771 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:30.772 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:30.772 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:30.772 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:30.772 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:30.774 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:30.774 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:30.774 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:30.774 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:30.774 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:30.775 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:30.775 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:30.775 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:30.775 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:30.775 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:30.776 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:30.776 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:30.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:30.776 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:30.776 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:30.776 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:30.777 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:30.777 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:30.778 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:17:35.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:35.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:35.780 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:35.822 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:35.822 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:35.873 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:35.873 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:35.874 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:35.874 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:35.874 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:35.874 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:35.875 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:35.875 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:35.875 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:35.875 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:35.875 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:35.876 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:35.876 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:35.876 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.545187_139.1", 2023-10-12T19:17:35.876 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:35.877 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:35.877 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:35.877 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.545187Z", 2023-10-12T19:17:35.877 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:35.878 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:35.878 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:35.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:35.878 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:35.878 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:35.879 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:35.879 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:35.879 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:35.879 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:35.879 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:35.880 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:35.880 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:35.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:35.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:35.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:35.881 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:35.881 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:35.881 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:35.881 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:35.881 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:35.882 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:35.882 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:35.882 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:35.882 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:35.882 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:35.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:35.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:35.883 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:35.883 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:35.883 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:35.884 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:35.884 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:35.884 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:35.884 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:35.885 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:35.885 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:35.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:35.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:35.886 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:35.886 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:35.886 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:35.886 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:35.887 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:35.887 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:35.887 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:35.888 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.545187_139.1 2023-10-12T19:17:40.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:40.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:40.891 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:40.953 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:40.953 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:41.061 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:41.061 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:17:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:17:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:41.074 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:41.075 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.436071_129.1 2023-10-12T19:17:46.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:46.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:46.077 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:46.142 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:46.142 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:46.497 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:46.498 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:46.498 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:46.498 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:46.498 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:46.499 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:46.499 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:46.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:46.499 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:46.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:46.500 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:46.500 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:46.500 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:46.500 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:17:46.500 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:46.501 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:46.501 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:46.501 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:17:46.501 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:46.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:46.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:46.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:46.502 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:46.502 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:46.502 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:46.503 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:46.503 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:46.503 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:46.503 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:46.503 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:46.503 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:46.504 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:46.504 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:46.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:46.504 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:46.504 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:46.505 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:46.505 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:46.505 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:46.505 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:46.505 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:46.506 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:46.506 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:46.506 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:46.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:46.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:46.507 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:46.507 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:46.507 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:46.507 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:46.507 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:46.508 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:46.508 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:46.508 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:46.508 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:46.508 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:46.509 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:46.509 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:46.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:46.509 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:46.509 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:46.510 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:46.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:46.510 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:46.510 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.470944_133.1 2023-10-12T19:17:51.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:51.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:51.514 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:51.596 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:51.596 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:51.913 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470052_132.1", 2023-10-12T19:17:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470052Z", 2023-10-12T19:17:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:51.919 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:51.919 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:51.919 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:51.919 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:51.919 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:51.921 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:51.921 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:51.921 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:51.921 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:51.921 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:51.922 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:51.922 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:51.922 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:51.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:51.922 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:51.923 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:51.923 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:51.923 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:51.923 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:51.923 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:51.923 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:51.924 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:51.924 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:51.924 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:51.924 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:51.924 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:51.925 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:51.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:51.925 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:51.925 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:51.925 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:51.926 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:51.926 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:51.927 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.470052_132.1 2023-10-12T19:17:56.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:56.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:17:56.929 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:17:57.011 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:17:57.012 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:17:57.066 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:17:57.067 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:17:57.067 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:17:57.068 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:17:57.068 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:17:57.068 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:17:57.068 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:57.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:57.069 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:17:57.069 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:57.069 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:17:57.070 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:57.070 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:57.070 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:17:57.071 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:57.071 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:17:57.071 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:57.071 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:17:57.071 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:57.072 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:57.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:57.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:57.072 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:17:57.072 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:57.072 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:57.073 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:17:57.073 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:57.073 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:57.073 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:57.073 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:17:57.074 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:57.074 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:57.074 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:57.074 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:57.074 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:17:57.075 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:57.075 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:57.075 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:57.076 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:57.076 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:57.076 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:57.076 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:57.077 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:57.077 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:57.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:57.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:17:57.077 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:17:57.077 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:17:57.078 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:17:57.078 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:17:57.078 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:17:57.078 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:17:57.078 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:17:57.079 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:17:57.079 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:17:57.079 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:57.079 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:57.079 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:17:57.080 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:17:57.080 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:17:57.080 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:17:57.080 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:17:57.081 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:17:57.081 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:17:57.082 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.470944_133.1 2023-10-12T19:18:02.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:02.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:02.085 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:02.170 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:02.171 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:02.228 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:02.229 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:02.229 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:02.229 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:02.229 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:02.230 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:02.230 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:02.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:02.230 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:02.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:02.231 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:02.231 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:02.231 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:02.231 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.495204_138.1", 2023-10-12T19:18:02.231 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:02.232 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:02.232 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:02.232 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.495204Z", 2023-10-12T19:18:02.232 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:02.232 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:02.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:02.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:02.233 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:02.233 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:02.233 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:02.234 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:02.234 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:02.234 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:02.234 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:02.235 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:02.235 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:02.235 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:02.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:02.235 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:02.236 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:02.236 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:02.236 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:02.236 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:02.237 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:02.237 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:02.237 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:02.237 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:02.237 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:02.238 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:02.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:02.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:02.238 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:02.238 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:02.239 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:02.241 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:02.241 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:02.241 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:02.242 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:02.242 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:02.242 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:02.242 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:02.243 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:02.243 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:02.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:02.243 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:02.243 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:02.244 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:02.244 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:02.244 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:02.245 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.495204_138.1 2023-10-12T19:18:07.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:07.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:07.249 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:07.294 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:07.294 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:07.349 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:07.349 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:07.349 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:07.350 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:07.350 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:07.351 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:07.351 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:07.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:07.351 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:07.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:07.352 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:07.353 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:07.353 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:07.353 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.495204_138.1", 2023-10-12T19:18:07.354 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:07.354 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:07.354 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:07.355 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.495204Z", 2023-10-12T19:18:07.355 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:07.355 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:07.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:07.356 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:07.356 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:07.356 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:07.356 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:07.356 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:07.357 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:07.357 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:07.357 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:07.357 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:07.357 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:07.358 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:07.358 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:07.358 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:07.358 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:07.359 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:07.359 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:07.359 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:07.359 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:07.359 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:07.359 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:07.360 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:07.360 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:07.360 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:07.360 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:07.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:07.361 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:07.361 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:07.361 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:07.361 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:07.361 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:07.362 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:07.362 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:07.362 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:07.362 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:07.362 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:07.363 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:07.363 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:07.363 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:07.363 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:07.363 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:07.363 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:07.364 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:07.364 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:07.365 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.495204_138.1 2023-10-12T19:18:12.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:12.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:12.367 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:12.409 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:12.410 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:12.460 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:12.461 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:12.461 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:12.461 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:12.461 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:12.461 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:12.462 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:12.462 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:12.462 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:12.462 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:12.463 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:12.463 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:12.463 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:12.463 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:18:12.463 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:12.463 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:18:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:12.464 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:12.465 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:12.466 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:12.466 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:12.466 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:12.466 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:12.466 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:12.467 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:12.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:12.467 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:12.467 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:12.467 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:12.468 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:12.468 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:12.468 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:12.468 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:12.468 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:12.468 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:12.469 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:12.469 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:12.469 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:12.469 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:12.469 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:12.470 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:12.470 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:12.470 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:12.470 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:12.470 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:12.471 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:12.471 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:12.471 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:12.471 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:12.471 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:12.471 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:12.472 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:12.472 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:12.472 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:12.472 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:12.472 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:12.473 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:12.474 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:18:17.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:17.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:17.476 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:17.518 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:17.519 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:17.569 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:17.570 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:17.570 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:17.570 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:17.570 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:17.571 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:17.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:17.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:17.571 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:17.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:17.571 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:17.572 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:17.572 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:17.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:18:17.572 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:17.572 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:17.573 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:17.573 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:18:17.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:17.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:17.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:17.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:17.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:17.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:17.574 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:17.574 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:17.574 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:17.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:17.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:17.575 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:17.575 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:17.575 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:17.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:17.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:17.576 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:17.576 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:17.576 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:17.576 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:17.577 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:17.577 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:17.577 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:17.577 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:17.577 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:17.577 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:17.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:17.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:17.578 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:17.578 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:17.578 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:17.579 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:17.579 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:17.579 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:17.579 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:17.579 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:17.580 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:17.580 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:17.580 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:17.580 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:17.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:17.580 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:17.581 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:17.581 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:17.581 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:17.581 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:17.582 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:18:22.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:22.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:22.585 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:22.685 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:22.685 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:22.837 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:22.838 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:22.838 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:22.838 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:22.838 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:22.839 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:18:22.840 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:18:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:22.841 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:22.842 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:22.843 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:22.843 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:22.843 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:22.843 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:22.843 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:22.844 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:22.844 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:22.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:22.844 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:22.844 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:22.845 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:22.845 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:22.845 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:22.845 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:22.845 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:22.846 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:22.846 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:22.846 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:22.846 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:22.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:22.847 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:22.847 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:22.847 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:22.847 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:22.847 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:22.848 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:22.848 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:22.848 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:22.848 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:22.848 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:22.849 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:22.849 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:22.849 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:22.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:22.849 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:22.850 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:22.850 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:22.850 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:22.850 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:22.851 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.470944_133.1 2023-10-12T19:18:27.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:27.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:27.854 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:27.896 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:27.897 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:27.951 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:27.951 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:27.952 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:27.953 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:18:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:27.954 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:18:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:27.955 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:27.956 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:27.957 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:27.958 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:27.959 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:27.960 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:27.960 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:27.960 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:27.960 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:27.960 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:27.961 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:27.961 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:27.961 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:27.961 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:27.961 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:27.961 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:27.962 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:27.962 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:27.962 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:27.962 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:27.962 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:27.963 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:27.963 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:27.963 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:27.963 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:27.963 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:27.964 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:27.965 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:18:32.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:32.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:32.967 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:33.033 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:33.033 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:33.498 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:33.498 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:33.498 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:33.499 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:33.499 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:33.499 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:33.499 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:33.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:33.499 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:33.500 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:33.500 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:33.500 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:33.500 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:33.500 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:18:33.501 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:33.501 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:33.501 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:33.501 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:18:33.501 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:33.502 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:33.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:33.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:33.502 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:33.502 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:33.503 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:33.503 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:33.503 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:33.503 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:33.503 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:33.504 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:33.504 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:33.504 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:33.504 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:33.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:33.505 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:33.505 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:33.505 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:33.505 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:33.505 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:33.505 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:33.506 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:33.506 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:33.506 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:33.506 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:33.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:33.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:33.507 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:33.507 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:33.507 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:33.507 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:33.508 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:33.508 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:33.508 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:33.508 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:33.508 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:33.509 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:33.509 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:33.509 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:33.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:33.509 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:33.510 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:33.510 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:33.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:33.510 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:33.511 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.432502_127.1 2023-10-12T19:18:38.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:38.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:38.513 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:38.603 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:38.604 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:38.882 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:38.883 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:38.883 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:38.883 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:38.884 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:38.884 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:38.884 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:38.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:38.884 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:38.884 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:38.885 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:38.885 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:38.885 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:38.885 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:18:38.885 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:38.886 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:38.886 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:38.886 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:18:38.886 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:38.887 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:38.887 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:38.887 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:38.887 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:38.887 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:38.888 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:38.888 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:38.888 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:38.888 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:38.888 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:38.889 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:38.889 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:38.889 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:38.889 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:38.889 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:38.889 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:38.890 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:38.890 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:38.890 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:38.890 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:38.891 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:38.891 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:38.891 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:38.891 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:38.891 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:38.892 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:38.892 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:38.892 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:38.892 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:38.892 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:38.892 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:38.893 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:38.893 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:38.893 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:38.893 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:38.893 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:38.894 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:38.894 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:38.894 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:38.894 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:38.894 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:38.894 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:38.895 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:38.895 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:38.895 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:38.896 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:18:43.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:43.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:43.899 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:43.982 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:43.982 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:44.270 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:44.270 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:44.270 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:44.270 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:44.271 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:44.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:44.272 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:44.272 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:44.272 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:44.272 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:18:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:18:44.273 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:44.274 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:44.275 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:44.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:44.277 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:44.278 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:44.278 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:44.278 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:44.278 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:44.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:44.279 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:44.280 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:44.281 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:44.281 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:44.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:44.281 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:44.282 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:44.282 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:44.282 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:44.282 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:44.283 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.436071_129.1 2023-10-12T19:18:49.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:49.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:49.285 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:49.329 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:49.329 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:49.391 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:49.391 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:49.392 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:49.393 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:49.394 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:49.394 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:18:49.394 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:49.394 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:49.395 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:49.395 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:18:49.395 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:49.395 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:49.395 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:49.396 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:49.396 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:49.396 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:49.396 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:49.396 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:49.397 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:49.397 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:49.397 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:49.397 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:49.397 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:49.398 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:49.398 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:49.398 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:49.398 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:49.399 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:49.399 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:49.399 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:49.399 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:49.400 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:49.400 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:49.400 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:49.400 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:49.400 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:49.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:49.401 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:49.401 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:49.401 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:49.401 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:49.402 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:49.402 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:49.402 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:49.402 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:49.402 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:49.403 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:49.403 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:49.403 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:49.403 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:49.403 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:49.404 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:49.404 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:49.404 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:49.404 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:49.404 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:49.405 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.407338_124.1 2023-10-12T19:18:54.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:54.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:54.408 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:54.451 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:54.451 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:54.504 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:54.504 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:54.504 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:54.504 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:54.504 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:54.505 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:54.505 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:54.505 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:54.505 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:54.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:54.505 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:54.506 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:54.506 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:54.506 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:18:54.506 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:54.506 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:54.507 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:54.507 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:18:54.507 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:54.507 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:54.507 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:54.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:54.508 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:54.508 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:54.508 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:54.508 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:54.508 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:54.509 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:54.509 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:54.509 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:54.509 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:54.509 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:54.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:54.510 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:54.510 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:54.510 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:54.510 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:54.510 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:54.511 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:54.511 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:54.511 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:54.511 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:54.511 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:54.512 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:54.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:54.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:54.512 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:54.512 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:54.512 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:54.513 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:54.513 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:54.513 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:54.513 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:54.513 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:54.514 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:54.514 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:54.514 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:54.514 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:54.514 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:54.514 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:54.515 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:54.515 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:54.515 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:54.515 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:54.516 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:18:59.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:59.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:18:59.519 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:18:59.561 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:18:59.561 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:18:59.613 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:18:59.613 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:18:59.613 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:18:59.613 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:18:59.614 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:18:59.614 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:18:59.614 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:59.614 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:59.614 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:18:59.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:59.615 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:18:59.615 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:59.615 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:59.615 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:18:59.616 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:59.616 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:18:59.616 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:59.616 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:18:59.616 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:59.617 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:59.617 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:59.617 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:59.617 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:18:59.617 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:59.618 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:59.618 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:18:59.618 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:59.618 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:59.618 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:59.619 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:18:59.619 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:59.619 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:59.619 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:59.619 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:59.619 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:18:59.620 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:59.620 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:59.620 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:59.620 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:59.620 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:59.621 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:59.621 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:59.621 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:59.621 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:59.621 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:59.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:18:59.622 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:18:59.622 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:18:59.625 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:18:59.625 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:18:59.625 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:18:59.625 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:18:59.625 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:18:59.626 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:18:59.626 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:18:59.626 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:59.626 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:59.626 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:18:59.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:18:59.627 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:18:59.627 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:18:59.627 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:18:59.627 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:18:59.628 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:18:59.629 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.432502_127.1 2023-10-12T19:19:04.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:04.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:04.632 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:04.750 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:04.751 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:04.882 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:04.882 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:04.882 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:04.883 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:04.883 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:04.883 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:04.883 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:04.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:04.884 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:04.884 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:04.884 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:04.884 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:04.884 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:04.885 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T19:19:04.885 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:04.885 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:04.885 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:04.885 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T19:19:04.885 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:04.886 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:04.886 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:04.886 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:04.886 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:04.886 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:04.887 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:04.887 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:04.887 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:04.887 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:04.887 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:04.888 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:04.888 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:04.888 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:04.888 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:04.888 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:04.889 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:04.889 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:04.889 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:04.889 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:04.889 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:04.890 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:04.890 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:04.890 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:04.890 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:04.890 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:04.891 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:04.891 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:04.891 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:04.891 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:04.891 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:04.891 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:04.892 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:04.892 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:04.892 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:04.892 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:04.892 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:04.893 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:04.893 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:04.893 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:04.893 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:04.893 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:04.893 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:04.894 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:04.894 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:04.894 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:04.895 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.438188_130.1 2023-10-12T19:19:09.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:09.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:09.897 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:09.978 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:09.979 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:10.270 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:10.270 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:10.270 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:10.270 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:10.271 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:10.271 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:10.271 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:10.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:10.271 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:10.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:10.272 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:10.272 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:10.272 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:10.272 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:19:10.273 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:10.273 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:10.273 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:10.273 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:19:10.274 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:10.274 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:10.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:10.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:10.275 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:10.275 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:10.275 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:10.276 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:10.276 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:10.276 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:10.277 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:10.277 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:10.278 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:10.278 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:10.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:10.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:10.279 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:10.279 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:10.279 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:10.280 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:10.280 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:10.280 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:10.281 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:10.281 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:10.281 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:10.282 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:10.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:10.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:10.282 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:10.282 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:10.283 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:10.283 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:10.283 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:10.283 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:10.284 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:10.284 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:10.284 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:10.284 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:10.284 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:10.285 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:10.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:10.285 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:10.285 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:10.285 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:10.286 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:10.286 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:10.287 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.470944_133.1 2023-10-12T19:19:15.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:15.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:15.289 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:15.376 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:15.377 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:15.504 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:15.504 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:15.505 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:15.506 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:15.507 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:15.507 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.574307_146.1", 2023-10-12T19:19:15.507 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:15.507 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:15.507 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:15.508 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.574307Z", 2023-10-12T19:19:15.508 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:15.508 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:15.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:15.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:15.508 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:15.509 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:15.509 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:15.509 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:15.509 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:15.509 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:15.510 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:15.510 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:15.510 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:15.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:15.510 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:15.511 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:15.511 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:15.511 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:15.511 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:15.511 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:15.511 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:15.512 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:15.512 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:15.512 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:15.512 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:15.512 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:15.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:15.513 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:15.513 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:15.513 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:15.513 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:15.513 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:15.513 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:15.514 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:15.514 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:15.514 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:15.514 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:15.514 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:15.515 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:15.515 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:15.515 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:15.515 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:15.515 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:15.515 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:15.516 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:15.516 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:15.517 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.574307_146.1 2023-10-12T19:19:20.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:20.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:20.519 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:20.577 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:20.577 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:20.998 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:20.999 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:20.999 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:20.999 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:21.000 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:21.000 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:21.000 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:21.000 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:21.000 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:21.001 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:21.001 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:21.001 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:21.001 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:21.001 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:19:21.002 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:21.002 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:21.002 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:21.002 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:19:21.002 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:21.003 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:21.003 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:21.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:21.003 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:21.003 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:21.004 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:21.004 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:21.004 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:21.004 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:21.004 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:21.005 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:21.005 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:21.005 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:21.005 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:21.005 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:21.005 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:21.006 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:21.006 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:21.006 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:21.006 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:21.007 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:21.007 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:21.007 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:21.007 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:21.007 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:21.008 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:21.008 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:21.008 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:21.008 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:21.008 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:21.009 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:21.009 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:21.009 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:21.009 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:21.009 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:21.010 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:21.010 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:21.010 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:21.010 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:21.011 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:21.011 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:21.011 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:21.011 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:21.011 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:21.012 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:21.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:19:26.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:26.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:26.015 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:26.061 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:26.062 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:26.116 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:26.117 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:26.117 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:26.117 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:26.117 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:26.118 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:26.118 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:26.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:26.118 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:26.119 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:26.119 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:26.119 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:26.119 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:26.119 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:19:26.119 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:26.120 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:26.120 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:26.120 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:19:26.120 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:26.120 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:26.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:26.121 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:26.121 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:26.121 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:26.121 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:26.122 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:26.122 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:26.122 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:26.122 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:26.122 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:26.123 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:26.123 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:26.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:26.123 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:26.123 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:26.124 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:26.124 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:26.124 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:26.124 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:26.124 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:26.124 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:26.125 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:26.125 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:26.125 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:26.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:26.125 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:26.126 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:26.126 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:26.126 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:26.126 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:26.126 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:26.127 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:26.127 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:26.127 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:26.127 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:26.127 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:26.128 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:26.128 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:26.128 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:26.128 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:26.129 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:26.129 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:26.129 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:26.129 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:26.130 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138058.331751_285.1 target=1_1697136751.408247_125.1 2023-10-12T19:19:31.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... FAIL 2023-10-12T19:19:31.133 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-46 2023-10-12T19:19:31.152 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-47 2023-10-12T19:19:31.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:31.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:31.335 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:31.418 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T19:19:31.430 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T19:19:31.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:31.431 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697138371.171804_293.1", 2023-10-12T19:19:31.431 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T19:19:31.171804Z" 2023-10-12T19:19:31.431 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:31.431 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:31.431 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:31.432 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T19:19:31.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:31.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:31.432 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:31.432 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:19:31.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:31.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:31.433 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:31.433 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:19:31.433 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:31.434 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T19:19:31.435 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T19:19:31.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:31.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:31.436 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:31.523 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:31.523 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:31.578 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:31.578 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:31.578 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:31.579 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:31.579 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:31.579 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:31.579 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:31.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:31.580 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:31.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:31.580 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:31.580 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:31.581 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:31.581 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.554239_141.1", 2023-10-12T19:19:31.581 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:31.581 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:31.581 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:31.582 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.554239Z", 2023-10-12T19:19:31.582 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:31.582 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:31.582 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:31.582 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:31.583 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:31.583 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:31.583 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:31.583 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:31.583 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:31.584 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:31.584 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:31.584 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:31.584 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:31.584 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:31.584 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:31.585 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:31.585 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:31.585 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:31.585 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:31.585 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:31.586 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:31.586 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:31.586 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:31.586 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:31.586 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:31.587 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:31.587 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:31.587 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:31.587 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:31.587 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:31.587 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:31.588 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:31.588 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:31.588 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:31.588 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:31.588 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:31.589 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:31.589 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:31.589 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:31.589 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:31.589 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:31.590 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:31.590 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:31.590 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:31.590 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:31.590 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:31.591 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.554239_141.1 2023-10-12T19:19:36.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:36.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:36.594 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:36.636 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:36.636 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:36.690 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:36.691 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:36.691 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:36.691 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:36.691 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:36.692 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:36.692 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:36.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:36.692 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:36.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:36.692 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:36.693 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:36.693 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:36.693 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:19:36.693 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:36.693 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:36.694 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:36.694 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:19:36.694 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:36.694 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:36.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:36.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:36.695 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:36.695 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:36.695 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:36.695 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:36.695 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:36.696 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:36.696 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:36.696 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:36.696 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:36.696 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:36.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:36.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:36.697 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:36.697 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:36.697 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:36.697 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:36.698 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:36.698 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:36.698 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:36.698 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:36.698 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:36.699 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:36.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:36.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:36.699 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:36.699 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:36.700 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:36.700 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:36.700 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:36.700 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:36.700 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:36.700 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:36.701 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:36.701 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:36.701 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:36.701 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:36.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:36.702 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:36.702 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:36.702 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:36.702 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:36.702 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:36.703 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470944_133.1 2023-10-12T19:19:41.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:41.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:41.706 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:41.753 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:41.753 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:41.809 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:41.809 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:41.811 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:41.811 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:41.811 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:41.811 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:41.811 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:19:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:19:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:41.821 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:41.821 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:41.821 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:41.821 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:41.821 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:41.822 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:41.823 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:19:46.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:46.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:46.824 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:46.867 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:46.867 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:46.918 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:19:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:19:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:46.931 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:46.932 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:19:51.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:51.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:51.934 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:51.993 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:51.993 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:52.049 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:52.049 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:52.049 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:52.050 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:52.050 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:52.050 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:52.051 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:52.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:52.051 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:52.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:52.051 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:52.052 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:52.052 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:52.052 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T19:19:52.052 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:52.052 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:52.053 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:52.053 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T19:19:52.053 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:52.054 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:52.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:52.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:52.054 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:52.055 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:52.055 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:52.055 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:52.055 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:52.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:52.057 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:52.058 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:52.058 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:52.058 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:52.058 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:52.059 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:52.059 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:52.059 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:52.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:52.059 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:52.060 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:52.060 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:52.060 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:52.060 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:52.063 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:52.063 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:52.063 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:52.064 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:52.064 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:52.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.473047_134.1 2023-10-12T19:19:57.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:57.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:19:57.067 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:19:57.114 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:19:57.114 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:19:57.202 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:19:57.202 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:19:57.202 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:19:57.203 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:19:57.203 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:19:57.203 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:19:57.203 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:57.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:57.203 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:19:57.204 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:57.204 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:19:57.204 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:57.204 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:57.204 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:19:57.205 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:57.205 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:19:57.205 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:57.205 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:19:57.205 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:57.206 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:57.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:57.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:57.206 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:19:57.206 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:57.207 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:57.207 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:19:57.207 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:57.207 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:57.207 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:57.208 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:19:57.208 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:57.208 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:57.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:57.209 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:57.209 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:19:57.209 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:57.209 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:57.209 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:57.209 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:57.210 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:57.210 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:57.210 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:57.210 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:57.210 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:57.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:57.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:19:57.211 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:19:57.211 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:19:57.211 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:19:57.212 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:19:57.212 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:19:57.212 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:19:57.212 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:19:57.212 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:19:57.213 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:19:57.213 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:57.213 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:57.213 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:19:57.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:19:57.213 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:19:57.214 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:19:57.214 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:19:57.214 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:19:57.214 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:19:57.215 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.410419_126.1 2023-10-12T19:20:02.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:02.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:02.217 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:02.260 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:02.261 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:02.321 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:02.321 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:02.321 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:02.322 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:02.322 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:02.322 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:02.322 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:02.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:02.323 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:02.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:02.323 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:02.323 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:02.324 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:02.324 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.586855_149.1", 2023-10-12T19:20:02.324 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:02.324 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:02.324 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:02.325 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.586855Z", 2023-10-12T19:20:02.325 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:02.325 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:02.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:02.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:02.326 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:02.326 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:02.326 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:02.326 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:02.327 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:02.327 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:02.327 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:02.327 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:02.327 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:02.328 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:02.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:02.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:02.328 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:02.328 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:02.329 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:02.329 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:02.329 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:02.329 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:02.329 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:02.330 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:02.330 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:02.330 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:02.330 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:02.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:02.331 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:02.331 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:02.331 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:02.331 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:02.331 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:02.332 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:02.332 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:02.332 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:02.332 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:02.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:02.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:02.333 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:02.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:02.333 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:02.333 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:02.334 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:02.334 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:02.334 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:02.335 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.586855_149.1 2023-10-12T19:20:07.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:07.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:07.338 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:07.446 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:07.446 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:07.641 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:07.641 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:07.642 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:07.642 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:07.642 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:07.642 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:07.643 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:07.643 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:07.643 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:07.643 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:07.643 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:07.644 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:07.644 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:07.644 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:20:07.644 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:07.644 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:07.645 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:07.645 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:20:07.645 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:07.645 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:07.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:07.646 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:07.646 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:07.646 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:07.646 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:07.646 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:07.647 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:07.647 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:07.647 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:07.647 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:07.647 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:07.648 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:07.648 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:07.648 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:07.648 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:07.648 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:07.648 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:07.649 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:07.649 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:07.649 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:07.649 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:07.649 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:07.650 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:07.650 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:07.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:07.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:07.650 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:07.651 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:07.651 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:07.651 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:07.651 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:07.651 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:07.652 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:07.652 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:07.652 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:07.652 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:07.652 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:07.652 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:07.653 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:07.653 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:07.653 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:07.653 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:07.653 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:07.654 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:07.655 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.435183_128.1 2023-10-12T19:20:12.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:12.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:12.657 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:12.743 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:12.744 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:12.874 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:12.874 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:12.874 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:12.875 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:12.875 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:12.875 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:12.875 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:12.875 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:12.876 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:12.876 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:12.876 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:12.876 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:12.876 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:12.877 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.583416_147.1", 2023-10-12T19:20:12.877 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:12.877 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:12.877 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:12.877 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.583416Z", 2023-10-12T19:20:12.878 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:12.878 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:12.878 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:12.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:12.878 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:12.879 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:12.879 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:12.879 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:12.879 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:12.879 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:12.880 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:12.880 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:12.880 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:12.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:12.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:12.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:12.881 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:12.881 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:12.881 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:12.881 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:12.881 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:12.882 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:12.882 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:12.882 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:12.882 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:12.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:12.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:12.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:12.883 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:12.883 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:12.884 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:12.884 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:12.884 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:12.884 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:12.884 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:12.885 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:12.885 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:12.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:12.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:12.885 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:12.886 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:12.886 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:12.886 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:12.886 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:12.886 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:12.887 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:12.888 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.583416_147.1 2023-10-12T19:20:17.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:17.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:17.890 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:17.968 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:17.969 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:18.436 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:18.437 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:18.437 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:18.437 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:18.438 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:18.438 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:18.438 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:18.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:18.439 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:18.439 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:18.440 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:18.440 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:18.440 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:18.441 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:20:18.441 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:18.442 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:18.442 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:18.442 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:20:18.443 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:18.443 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:18.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:18.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:18.444 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:18.444 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:18.444 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:18.444 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:18.444 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:18.445 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:18.445 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:18.445 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:18.445 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:18.445 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:18.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:18.446 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:18.446 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:18.446 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:18.446 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:18.447 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:18.447 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:18.447 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:18.447 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:18.447 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:18.448 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:18.448 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:18.448 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:18.448 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:18.448 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:18.449 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:18.449 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:18.449 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:18.449 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:18.449 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:18.449 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:18.450 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:18.450 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:18.450 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:18.450 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:18.451 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:18.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:18.451 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:18.451 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:18.451 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:18.452 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:18.452 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:18.453 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:20:23.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:23.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:23.456 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:23.505 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:23.505 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:23.571 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:23.571 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:23.572 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:23.572 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:23.572 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:23.572 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:23.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:23.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:23.573 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:23.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:23.573 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:23.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:23.574 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:23.574 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:20:23.574 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:23.574 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:23.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:23.575 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:20:23.575 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:23.575 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:23.575 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:23.577 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:23.577 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:23.577 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:23.578 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:23.578 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:23.578 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:23.578 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:23.578 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:23.579 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:23.579 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:23.579 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:23.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:23.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:23.580 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:23.580 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:23.580 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:23.580 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:23.580 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:23.581 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:23.581 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:23.581 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:23.581 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:23.581 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:23.582 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:23.582 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:23.582 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:23.582 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:23.582 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:23.583 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:23.583 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:23.583 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:23.583 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:23.583 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:23.584 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:23.584 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:23.584 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:23.584 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:23.584 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:23.584 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:23.585 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:23.585 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:23.585 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:23.585 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:23.586 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:20:28.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:28.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:28.589 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:28.632 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:28.632 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:28.687 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:28.687 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:28.687 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:28.688 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:28.688 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:28.688 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:28.688 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:28.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:28.689 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:28.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:28.689 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:28.689 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:28.689 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:28.690 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.488676_136.1", 2023-10-12T19:20:28.690 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:28.690 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:28.690 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:28.690 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.488676Z", 2023-10-12T19:20:28.691 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:28.691 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:28.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:28.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:28.691 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:28.691 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:28.692 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:28.692 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:28.692 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:28.692 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:28.692 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:28.693 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:28.693 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:28.693 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:28.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:28.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:28.694 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:28.694 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:28.694 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:28.694 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:28.694 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:28.695 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:28.695 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:28.695 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:28.695 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:28.695 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:28.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:28.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:28.696 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:28.696 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:28.696 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:28.696 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:28.697 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:28.697 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:28.697 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:28.697 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:28.697 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:28.697 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:28.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:28.698 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:28.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:28.698 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:28.699 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:28.699 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:28.699 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:28.699 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:28.702 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.488676_136.1 2023-10-12T19:20:33.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:33.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:33.704 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:33.746 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:33.746 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:33.829 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:33.829 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:33.829 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:33.830 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:33.830 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:33.830 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:33.830 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:33.830 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:33.831 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:33.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:33.831 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:33.831 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:33.831 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:33.832 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:20:33.832 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:33.832 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:33.832 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:33.832 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:20:33.833 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:33.833 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:33.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:33.833 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:33.833 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:33.834 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:33.834 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:33.834 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:33.834 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:33.834 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:33.835 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:33.835 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:33.835 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:33.835 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:33.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:33.835 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:33.836 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:33.836 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:33.836 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:33.836 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:33.836 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:33.837 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:33.837 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:33.837 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:33.837 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:33.837 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:33.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:33.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:33.838 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:33.838 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:33.838 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:33.839 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:33.839 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:33.839 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:33.841 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:33.841 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:33.842 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:33.842 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:33.842 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:33.844 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:33.845 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.407338_124.1 2023-10-12T19:20:38.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:38.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:38.847 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:38.939 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:38.940 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:39.227 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:39.227 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:39.227 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:39.228 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:39.229 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:20:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:39.230 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:20:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:39.231 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:39.232 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:39.233 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:39.233 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:39.233 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:39.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:39.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:39.234 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:39.234 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:39.234 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:39.234 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:39.234 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:39.235 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:39.235 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:39.235 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:39.235 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:39.235 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:39.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:39.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:39.236 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:39.236 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:39.236 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:39.236 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:39.237 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:39.237 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:39.237 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:39.237 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:39.237 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:39.238 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:39.238 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:39.238 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:39.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:39.238 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:39.238 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:39.239 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:39.239 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:39.239 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:39.240 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:20:44.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:44.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:44.243 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:44.323 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:44.323 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:44.667 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.488676_136.1", 2023-10-12T19:20:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.488676Z", 2023-10-12T19:20:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:44.675 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:44.675 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:44.675 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:44.675 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:44.675 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:44.679 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:44.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.488676_136.1 2023-10-12T19:20:49.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:49.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:49.683 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:49.771 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:49.771 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:49.871 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:49.872 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:49.872 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:49.872 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:49.872 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:49.872 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:49.873 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:49.873 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:49.873 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:49.873 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:49.873 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:49.874 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:49.874 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:49.874 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:20:49.874 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:49.874 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:49.875 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:49.875 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:20:49.875 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:49.875 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:49.875 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:49.876 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:49.876 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:49.876 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:49.876 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:49.876 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:49.877 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:49.877 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:49.877 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:49.877 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:49.877 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:49.878 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:49.878 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:49.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:49.878 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:49.879 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:49.879 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:49.879 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:49.879 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:49.880 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:49.880 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:49.880 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:49.880 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:49.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:49.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:49.881 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:49.881 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:49.881 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:49.882 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:49.882 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:49.882 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:49.882 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:49.882 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:49.883 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:49.883 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:49.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:49.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:49.883 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:49.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:49.884 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:49.884 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:49.884 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:49.884 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:49.884 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:49.886 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.435183_128.1 2023-10-12T19:20:54.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:54.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:20:54.888 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:20:54.982 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:20:54.983 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:20:55.204 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:20:55.205 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:20:55.205 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:20:55.205 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:20:55.205 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:20:55.205 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:20:55.206 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:55.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:55.206 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:20:55.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:55.206 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:20:55.207 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:55.207 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:55.207 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:20:55.207 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:55.207 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:20:55.208 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:55.208 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:20:55.208 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:55.208 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:55.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:55.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:55.209 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:20:55.209 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:55.209 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:55.209 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:20:55.209 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:55.210 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:55.210 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:55.210 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:20:55.210 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:55.210 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:55.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:55.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:55.211 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:20:55.211 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:55.211 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:55.212 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:55.212 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:55.212 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:55.212 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:55.212 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:55.212 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:55.213 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:55.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:55.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:20:55.213 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:20:55.213 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:20:55.214 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:20:55.214 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:20:55.214 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:20:55.214 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:20:55.214 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:20:55.215 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:20:55.215 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:20:55.215 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:55.215 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:55.215 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:20:55.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:20:55.216 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:20:55.216 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:20:55.216 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:20:55.216 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:20:55.216 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:20:55.217 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470944_133.1 2023-10-12T19:21:00.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:00.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:00.220 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:00.265 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:00.265 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:00.320 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:00.320 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:00.320 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:00.321 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:00.321 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:00.321 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:00.321 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:00.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:00.322 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:00.322 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:00.322 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:00.322 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:00.322 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:00.323 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:21:00.323 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:00.323 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:00.323 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:00.323 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:21:00.324 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:00.324 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:00.324 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:00.324 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:00.324 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:00.325 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:00.325 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:00.325 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:00.325 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:00.325 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:00.326 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:00.326 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:00.326 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:00.326 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:00.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:00.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:00.327 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:00.327 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:00.327 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:00.327 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:00.327 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:00.328 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:00.328 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:00.328 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:00.328 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:00.328 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:00.329 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:00.329 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:00.329 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:00.329 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:00.329 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:00.330 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:00.330 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:00.330 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:00.330 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:00.330 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:00.330 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:00.331 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:00.331 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:00.331 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:00.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:00.331 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:00.332 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:00.332 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:00.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:00.332 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:00.333 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470944_133.1 2023-10-12T19:21:05.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:05.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:05.335 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:05.378 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:05.378 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:05.430 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:05.430 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:05.431 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:05.431 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:05.431 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:05.431 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:05.431 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:05.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:05.432 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:05.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:05.432 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:05.432 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:05.433 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:05.433 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:21:05.433 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:05.433 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:05.433 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:05.434 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:21:05.434 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:05.434 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:05.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:05.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:05.435 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:05.435 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:05.435 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:05.435 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:05.435 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:05.435 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:05.436 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:05.436 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:05.436 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:05.436 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:05.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:05.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:05.437 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:05.437 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:05.437 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:05.437 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:05.438 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:05.438 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:05.438 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:05.438 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:05.438 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:05.439 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:05.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:05.439 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:05.439 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:05.439 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:05.440 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:05.440 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:05.440 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:05.440 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:05.440 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:05.440 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:05.441 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:05.441 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:05.441 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:05.441 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:05.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:05.441 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:05.442 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:05.442 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:05.442 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:05.442 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:05.443 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.436071_129.1 2023-10-12T19:21:10.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:10.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:10.446 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:10.488 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:10.488 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:10.540 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:10.540 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:10.540 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:10.540 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:10.541 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:10.541 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:10.541 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:10.541 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:10.541 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:10.542 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:10.542 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:10.542 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:10.542 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:10.542 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:21:10.543 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:10.543 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:10.543 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:10.543 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:21:10.543 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:10.544 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:10.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:10.544 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:10.544 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:10.544 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:10.544 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:10.545 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:10.545 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:10.545 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:10.545 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:10.545 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:10.546 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:10.546 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:10.546 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:10.546 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:10.546 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:10.547 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:10.547 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:10.547 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:10.547 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:10.547 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:10.548 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:10.548 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:10.548 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:10.548 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:10.548 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:10.549 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:10.549 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:10.549 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:10.549 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:10.549 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:10.550 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:10.550 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:10.550 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:10.550 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:10.550 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:10.551 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:10.551 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:10.551 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:10.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:10.551 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:10.551 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:10.552 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:10.552 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:10.552 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:10.553 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.436071_129.1 2023-10-12T19:21:15.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:15.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:15.556 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:15.632 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:15.632 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:15.832 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:15.833 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:15.833 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:15.833 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:15.833 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:15.834 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:15.834 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:15.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:15.834 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:15.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:15.835 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:15.835 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:15.835 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:15.835 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.492727_137.1", 2023-10-12T19:21:15.835 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:15.836 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:15.836 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:15.836 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.492727Z", 2023-10-12T19:21:15.836 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:15.836 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:15.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:15.837 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:15.837 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:15.837 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:15.837 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:15.837 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:15.838 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:15.838 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:15.838 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:15.838 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:15.838 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:15.839 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:15.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:15.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:15.839 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:15.839 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:15.840 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:15.840 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:15.840 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:15.840 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:15.840 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:15.840 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:15.841 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:15.841 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:15.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:15.841 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:15.841 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:15.842 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:15.842 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:15.842 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:15.842 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:15.842 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:15.843 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:15.843 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:15.843 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:15.843 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:15.843 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:15.844 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:15.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:15.844 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:15.844 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:15.844 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:15.845 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:15.845 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:15.845 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.492727_137.1 2023-10-12T19:21:20.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:20.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:20.848 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:20.911 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:20.912 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:21.321 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:21.322 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:21.322 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:21.322 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:21.323 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:21.323 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:21.323 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:21.323 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:21.323 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:21:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:21:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:21.334 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:21.334 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:21.335 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.407338_124.1 2023-10-12T19:21:26.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:26.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:26.338 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:26.412 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:26.413 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:26.469 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:26.469 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:26.469 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:26.469 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:26.470 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:26.470 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:26.470 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:26.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:26.470 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:26.471 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:26.471 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:26.471 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:26.471 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:26.471 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.492727_137.1", 2023-10-12T19:21:26.472 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:26.472 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:26.472 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:26.472 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.492727Z", 2023-10-12T19:21:26.472 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:26.473 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:26.473 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:26.473 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:26.473 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:26.473 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:26.474 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:26.474 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:26.474 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:26.475 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:26.475 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:26.475 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:26.476 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:26.476 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:26.476 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:26.477 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:26.478 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:26.478 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:26.478 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:26.478 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:26.478 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:26.479 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:26.479 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:26.479 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:26.479 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:26.479 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:26.480 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:26.480 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:26.480 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:26.480 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:26.480 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:26.480 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:26.481 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:26.481 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:26.481 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:26.481 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:26.481 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:26.482 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:26.482 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:26.482 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:26.482 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:26.482 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:26.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.492727_137.1 2023-10-12T19:21:31.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:31.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:31.486 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:31.593 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:31.594 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:31.858 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:31.858 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:31.858 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:31.859 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:31.859 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:31.859 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:31.859 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:31.859 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:31.860 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:31.860 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:31.860 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:31.860 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:31.860 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:31.861 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.492727_137.1", 2023-10-12T19:21:31.861 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:31.861 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:31.861 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:31.861 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.492727Z", 2023-10-12T19:21:31.862 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:31.862 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:31.862 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:31.862 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:31.862 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:31.863 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:31.863 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:31.863 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:31.863 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:31.863 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:31.864 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:31.864 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:31.864 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:31.864 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:31.864 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:31.865 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:31.865 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:31.865 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:31.865 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:31.865 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:31.866 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:31.866 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:31.866 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:31.866 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:31.866 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:31.867 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:31.867 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:31.867 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:31.867 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:31.867 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:31.868 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:31.868 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:31.868 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:31.868 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:31.868 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:31.868 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:31.869 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:31.869 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:31.869 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:31.869 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:31.869 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:31.870 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:31.870 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:31.870 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:31.870 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:31.870 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:31.872 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.492727_137.1 2023-10-12T19:21:36.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:36.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:36.875 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:36.960 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:36.961 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:37.405 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:37.405 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:37.406 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:37.406 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:37.406 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:37.406 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:37.406 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:37.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:37.407 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:37.407 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:37.407 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:37.407 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:37.408 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:37.408 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:21:37.408 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:37.408 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:37.408 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:37.409 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:21:37.409 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:37.409 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:37.409 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:37.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:37.410 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:37.410 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:37.410 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:37.410 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:37.410 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:37.411 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:37.411 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:37.411 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:37.411 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:37.411 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:37.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:37.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:37.412 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:37.412 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:37.412 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:37.412 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:37.413 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:37.413 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:37.413 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:37.413 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:37.413 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:37.414 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:37.414 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:37.414 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:37.414 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:37.414 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:37.415 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:37.415 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:37.415 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:37.416 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:37.416 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:37.416 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:37.416 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:37.416 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:37.416 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:37.417 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:37.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:37.417 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:37.417 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:37.418 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:37.418 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:37.418 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:37.419 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.435183_128.1 2023-10-12T19:21:42.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:42.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:42.421 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:42.464 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:42.464 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:42.516 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:42.517 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:42.517 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:42.517 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:42.517 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.574307_146.1", 2023-10-12T19:21:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.574307Z", 2023-10-12T19:21:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:42.522 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:42.522 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:42.522 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:42.522 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:42.522 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:42.529 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:42.530 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.574307_146.1 2023-10-12T19:21:47.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:47.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:47.533 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:47.576 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:47.576 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:47.627 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:47.627 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:47.628 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:47.628 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:47.628 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:47.628 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:47.628 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:47.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:47.629 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:47.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:47.629 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:47.630 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:47.630 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:47.630 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:21:47.630 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:47.630 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:47.631 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:47.631 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:21:47.631 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:47.631 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:47.631 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:47.632 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:47.632 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:47.632 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:47.632 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:47.632 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:47.633 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:47.633 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:47.633 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:47.633 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:47.633 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:47.634 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:47.634 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:47.634 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:47.634 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:47.634 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:47.635 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:47.635 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:47.635 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:47.635 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:47.636 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:47.636 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:47.636 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:47.636 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:47.636 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:47.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:47.637 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:47.637 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:47.637 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:47.637 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:47.638 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:47.638 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:47.638 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:47.638 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:47.638 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:47.639 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:47.639 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:47.639 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:47.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:47.639 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:47.640 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:47.640 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:47.640 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:47.640 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:47.641 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.407338_124.1 2023-10-12T19:21:52.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:52.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:52.644 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:52.687 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:52.687 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:52.739 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:52.739 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:52.739 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:52.739 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:52.740 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:52.740 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:52.740 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:52.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:52.741 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:52.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:52.741 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:52.742 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:52.742 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:52.742 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:21:52.742 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:52.743 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:52.743 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:52.743 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:21:52.743 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:52.743 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:52.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:52.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:52.744 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:52.744 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:52.744 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:52.744 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:52.745 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:52.745 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:52.745 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:52.745 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:52.745 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:52.746 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:52.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:52.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:52.746 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:52.746 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:52.747 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:52.747 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:52.747 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:52.747 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:52.747 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:52.748 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:52.748 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:52.748 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:52.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:52.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:52.749 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:52.749 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:52.749 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:52.749 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:52.749 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:52.750 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:52.750 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:52.750 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:52.750 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:52.751 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:52.751 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:52.751 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:52.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:52.752 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:52.752 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:52.752 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:52.752 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:52.752 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:52.754 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.407338_124.1 2023-10-12T19:21:57.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:57.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:21:57.756 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:21:57.860 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:21:57.860 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:21:58.137 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:21:58.137 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:21:58.137 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:21:58.137 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:21:58.138 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:21:58.138 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:21:58.138 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:58.138 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:58.138 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:21:58.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:58.139 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:21:58.139 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:58.139 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:58.139 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T19:21:58.140 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:58.140 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:21:58.140 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:58.140 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T19:21:58.140 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:58.141 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:58.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:58.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:58.141 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:21:58.141 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:58.142 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:58.142 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:21:58.142 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:58.142 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:58.142 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:58.143 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:21:58.143 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:58.143 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:58.143 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:58.143 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:21:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:58.145 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:58.145 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:58.145 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:58.145 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:58.145 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:58.146 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:58.146 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:21:58.146 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:21:58.146 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:21:58.146 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:21:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:21:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:21:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:21:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:21:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:21:58.148 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:21:58.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:58.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:58.148 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:21:58.148 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:21:58.149 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:21:58.149 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:21:58.149 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:21:58.149 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:21:58.149 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:21:58.151 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.473047_134.1 2023-10-12T19:22:03.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:03.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:03.153 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:03.196 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:03.196 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:03.249 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:03.250 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:03.250 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:03.250 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:03.251 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:03.251 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:03.251 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:03.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:03.251 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:03.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:03.252 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:03.252 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:03.252 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:03.252 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T19:22:03.253 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:03.253 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:03.253 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:03.253 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T19:22:03.253 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:03.254 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:03.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:03.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:03.254 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:03.254 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:03.255 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:03.255 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:03.255 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:03.255 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:03.256 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:03.257 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:03.257 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:03.257 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:03.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:03.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:03.258 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:03.258 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:03.258 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:03.258 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:03.258 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:03.259 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:03.259 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:03.259 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:03.259 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:03.259 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:03.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:03.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:03.260 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:03.260 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:03.260 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:03.260 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:03.261 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:03.261 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:03.261 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:03.261 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:03.261 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:03.262 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:03.262 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:03.262 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:03.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:03.262 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:03.263 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:03.263 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:03.263 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:03.263 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:03.264 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.438188_130.1 2023-10-12T19:22:08.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:08.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:08.267 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:08.355 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:08.355 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:08.773 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:22:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:22:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:08.785 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:08.786 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:22:13.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:13.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:13.788 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:13.849 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:13.850 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:14.225 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:14.225 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:14.226 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:14.226 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:14.226 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:14.226 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:14.227 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:14.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:14.227 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:14.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:14.228 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:14.228 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:14.228 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:14.228 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:22:14.228 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:14.229 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:14.229 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:14.229 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:22:14.229 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:14.229 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:14.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:14.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:14.230 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:14.230 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:14.230 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:14.231 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:14.231 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:14.231 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:14.232 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:14.232 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:14.232 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:14.232 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:14.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:14.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:14.233 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:14.233 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:14.233 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:14.234 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:14.234 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:14.234 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:14.234 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:14.234 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:14.235 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:14.235 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:14.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:14.235 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:14.235 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:14.236 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:14.236 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:14.236 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:14.236 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:14.236 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:14.237 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:14.237 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:14.237 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:14.237 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:14.237 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:14.238 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:14.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:14.238 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:14.238 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:14.238 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:14.239 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:14.239 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:14.240 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:22:19.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:19.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:19.244 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:19.332 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:19.332 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:19.548 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:19.549 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:19.549 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:19.549 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:19.549 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:19.550 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:19.550 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:19.550 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:19.550 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:19.550 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:19.551 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:19.551 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:19.551 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:19.551 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:22:19.551 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:19.552 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:19.552 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:19.552 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:22:19.552 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:19.552 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:19.552 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:19.553 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:19.553 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:19.553 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:19.553 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:19.553 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:19.554 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:19.554 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:19.554 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:19.554 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:19.554 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:19.555 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:19.555 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:19.555 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:19.555 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:19.555 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:19.556 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:19.556 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:19.556 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:19.556 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:19.556 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:19.556 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:19.557 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:19.557 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:19.557 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:19.557 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:19.557 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:19.558 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:19.558 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:19.558 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:19.558 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:19.558 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:19.559 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:19.559 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:19.559 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:19.559 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:19.559 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:19.560 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:19.560 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:19.560 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:19.560 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:19.560 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:19.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:19.561 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:19.562 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:22:24.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:24.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:24.564 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:24.655 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:24.656 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:24.817 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.611004_153.1", 2023-10-12T19:22:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.611004Z", 2023-10-12T19:22:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:24.831 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:24.832 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.611004_153.1 2023-10-12T19:22:29.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:29.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:29.834 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:29.932 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:29.933 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:30.324 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:30.325 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:30.325 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:30.325 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:30.326 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:30.326 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:30.326 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:30.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:30.326 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:30.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:30.327 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:30.327 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:30.327 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:30.327 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:22:30.327 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:30.328 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:30.328 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:30.328 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:22:30.328 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:30.328 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:30.329 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:30.329 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:30.329 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:30.329 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:30.329 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:30.330 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:30.330 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:30.330 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:30.330 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:30.330 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:30.331 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:30.331 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:30.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:30.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:30.331 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:30.332 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:30.332 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:30.332 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:30.332 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:30.332 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:30.333 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:30.333 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:30.333 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:30.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:30.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:30.333 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:30.334 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:30.334 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:30.334 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:30.334 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:30.334 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:30.335 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:30.335 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:30.335 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:30.335 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:30.335 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:30.336 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:30.336 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:30.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:30.336 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:30.336 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:30.337 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:30.337 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:30.337 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:30.338 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.407338_124.1 2023-10-12T19:22:35.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:35.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:35.341 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:35.447 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:35.448 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:35.676 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:35.677 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:35.677 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:35.677 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:35.677 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:35.678 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:35.678 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:35.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:35.678 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:35.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:35.679 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:35.679 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:35.679 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:35.679 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:22:35.679 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:35.680 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:35.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:35.680 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:22:35.680 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:35.680 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:35.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:35.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:35.681 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:35.681 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:35.681 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:35.682 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:35.682 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:35.682 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:35.682 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:35.682 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:35.683 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:35.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:35.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:35.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:35.684 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:35.684 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:35.684 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:35.684 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:35.684 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:35.685 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:35.685 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:35.685 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:35.685 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:35.685 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:35.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:35.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:35.686 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:35.686 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:35.686 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:35.687 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:35.687 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:35.687 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:35.687 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:35.687 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:35.688 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:35.688 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:35.688 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:35.688 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:35.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:35.688 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:35.689 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:35.689 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:35.689 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:35.689 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:35.690 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:22:40.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:40.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:40.693 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:40.778 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:40.778 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:40.991 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:40.992 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:40.992 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:40.993 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:40.993 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:40.993 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:40.994 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:40.994 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:40.994 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:40.995 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:40.995 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:40.995 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:40.995 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:22:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:22:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:41.001 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:41.001 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:41.001 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:41.001 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:41.002 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:41.002 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:41.002 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:41.002 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:41.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:41.003 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:41.003 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:41.003 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:41.003 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:41.005 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:41.005 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:41.005 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:41.005 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:41.006 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:41.006 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:41.006 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:41.007 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:41.009 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:41.011 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:22:46.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:46.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:46.013 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:46.104 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:46.104 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:46.482 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:46.483 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:46.483 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:46.483 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:46.483 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:46.483 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:46.484 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:46.484 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:46.484 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:46.484 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:46.484 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:46.485 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:46.485 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:46.485 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:22:46.485 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:46.485 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:46.485 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:46.486 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:22:46.486 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:46.486 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:46.486 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:46.486 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:46.487 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:46.487 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:46.487 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:46.487 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:46.487 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:46.488 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:46.488 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:46.488 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:46.488 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:46.488 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:46.489 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:46.489 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:46.489 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:46.489 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:46.489 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:46.490 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:46.490 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:46.490 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:46.491 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:46.491 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:46.491 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:46.491 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:46.491 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:46.492 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:46.492 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:46.492 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:46.492 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:46.492 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:46.493 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:46.493 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:46.493 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:46.493 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:46.493 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:46.494 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:46.494 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:46.494 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:46.494 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:46.494 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:46.494 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:46.495 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:46.495 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:46.495 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:46.495 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.436071_129.1 2023-10-12T19:22:51.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:51.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:51.498 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:51.549 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:51.549 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:51.903 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:51.903 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:51.904 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:51.904 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:51.904 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:51.905 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:51.905 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:51.905 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:51.906 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:51.906 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:51.907 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:51.907 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:51.907 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:51.908 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:22:51.908 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:51.908 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:22:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:51.910 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:51.910 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:51.911 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:51.911 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:51.912 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:51.913 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:51.917 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:51.918 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:51.919 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:51.919 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:51.919 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:51.920 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:51.921 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:51.921 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:51.921 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:51.922 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:51.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:51.922 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:51.924 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:51.924 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:51.924 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:51.925 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:51.925 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.408247_125.1 2023-10-12T19:22:56.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:56.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:22:56.929 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:22:56.973 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:22:56.973 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:22:57.028 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:22:57.028 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:22:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:22:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:22:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:22:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:22:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:57.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:57.030 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:22:57.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:57.030 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:22:57.030 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:57.031 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:57.031 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:22:57.031 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:57.031 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:22:57.031 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:57.032 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:22:57.032 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:57.032 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:57.032 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:57.032 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:57.033 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:22:57.033 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:57.033 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:57.033 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:22:57.033 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:57.034 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:57.034 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:57.034 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:22:57.034 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:57.034 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:57.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:57.035 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:57.035 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:22:57.035 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:57.035 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:57.035 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:57.036 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:57.036 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:57.036 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:57.036 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:57.036 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:57.037 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:57.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:57.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:22:57.037 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:22:57.037 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:22:57.038 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:22:57.038 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:22:57.038 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:22:57.038 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:22:57.038 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:22:57.038 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:22:57.039 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:22:57.039 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:57.039 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:57.039 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:22:57.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:22:57.040 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:22:57.040 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:22:57.040 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:22:57.040 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:22:57.040 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:22:57.041 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.407338_124.1 2023-10-12T19:23:02.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:02.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:02.043 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:02.087 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:02.087 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:02.140 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:02.140 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:02.141 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:02.141 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:02.141 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:02.141 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:02.141 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:02.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:02.142 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:02.142 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:02.142 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:02.142 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:02.143 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:02.143 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470052_132.1", 2023-10-12T19:23:02.143 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:02.143 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:02.143 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:02.144 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470052Z", 2023-10-12T19:23:02.144 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:02.144 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:02.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:02.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:02.144 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:02.145 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:02.145 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:02.145 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:02.145 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:02.145 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:02.146 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:02.146 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:02.146 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:02.146 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:02.146 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:02.147 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:02.147 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:02.147 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:02.147 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:02.147 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:02.148 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:02.148 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:02.148 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:02.148 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:02.148 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:02.149 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:02.149 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:02.149 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:02.149 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:02.149 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:02.150 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:02.150 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:02.150 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:02.150 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:02.150 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:02.151 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:02.151 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:02.151 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:02.151 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:02.151 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:02.151 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:02.152 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:02.152 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:02.152 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:02.152 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:02.152 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:02.154 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470052_132.1 2023-10-12T19:23:07.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:07.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:07.156 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:07.219 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:07.220 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:07.278 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:07.281 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:07.281 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:07.281 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:07.281 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:23:07.281 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:07.282 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:07.282 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:07.282 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:23:07.282 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:07.283 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:07.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:07.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:07.283 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:07.284 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:07.284 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:07.284 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:07.284 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:07.292 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:07.293 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470944_133.1 2023-10-12T19:23:12.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:12.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:12.296 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:12.374 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:12.375 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:12.483 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:12.484 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:12.484 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:12.484 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:12.484 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:12.484 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:12.485 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:12.485 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:12.485 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:12.485 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:12.485 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:12.486 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:12.486 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:12.486 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.554239_141.1", 2023-10-12T19:23:12.486 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:12.486 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:12.487 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:12.487 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.554239Z", 2023-10-12T19:23:12.487 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:12.487 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:12.487 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:12.488 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:12.488 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:12.488 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:12.488 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:12.488 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:12.489 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:12.489 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:12.489 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:12.489 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:12.489 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:12.490 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:12.490 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:12.490 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:12.490 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:12.490 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:12.491 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:12.491 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:12.491 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:12.491 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:12.491 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:12.492 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:12.492 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:12.492 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:12.492 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:12.492 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:12.493 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:12.493 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:12.493 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:12.493 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:12.493 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:12.494 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:12.494 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:12.494 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:12.494 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:12.494 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:12.495 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:12.495 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:12.495 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:12.495 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:12.495 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:12.496 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:12.496 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:12.496 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:12.497 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.554239_141.1 2023-10-12T19:23:17.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:17.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:17.499 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:17.578 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:17.578 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:17.908 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:17.909 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:17.909 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:17.909 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:17.909 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:17.909 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:17.910 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:17.910 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:17.910 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:17.910 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:17.910 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:17.911 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:17.911 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:17.911 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:23:17.911 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:17.911 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:17.912 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:17.912 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:23:17.912 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:17.912 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:17.912 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:17.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:17.913 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:17.913 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:17.913 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:17.913 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:17.913 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:17.914 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:17.914 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:17.914 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:17.914 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:17.914 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:17.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:17.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:17.915 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:17.915 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:17.915 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:17.916 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:17.916 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:17.916 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:17.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:17.916 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:17.916 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:17.917 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:17.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:17.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:17.917 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:17.917 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:17.918 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:17.918 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:17.918 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:17.918 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:17.918 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:17.919 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:17.919 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:17.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:17.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:17.919 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:17.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:17.920 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:17.920 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:17.920 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:17.920 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:17.921 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:17.921 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470944_133.1 2023-10-12T19:23:22.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:22.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:22.923 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:22.968 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:22.968 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:23.085 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:23.085 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:23.085 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:23.085 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:23.086 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:23.086 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:23.086 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:23.086 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:23.086 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:23.087 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:23.087 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:23.087 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:23.087 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:23.087 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:23:23.088 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:23.088 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:23.088 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:23.088 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:23:23.088 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:23.089 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:23.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:23.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:23.089 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:23.089 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:23.090 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:23.090 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:23.090 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:23.091 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:23.091 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:23.091 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:23.091 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:23.091 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:23.092 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:23.092 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:23.092 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:23.092 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:23.092 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:23.093 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:23.093 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:23.093 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:23.093 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:23.093 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:23.094 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:23.094 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:23.094 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:23.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:23.094 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:23.095 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:23.095 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:23.095 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:23.095 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:23.095 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:23.096 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:23.096 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:23.096 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:23.096 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:23.097 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:23.097 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:23.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:23.097 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:23.097 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:23.098 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:23.098 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:23.098 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:23.099 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470944_133.1 2023-10-12T19:23:28.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:28.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:28.101 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:28.144 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:28.144 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:28.200 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:28.201 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:28.201 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:28.201 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:28.201 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:28.201 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:28.202 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:28.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:28.202 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:28.202 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:28.202 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:28.203 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:28.203 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:28.203 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.492727_137.1", 2023-10-12T19:23:28.203 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:28.203 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:28.204 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:28.204 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.492727Z", 2023-10-12T19:23:28.204 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:28.204 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:28.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:28.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:28.205 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:28.205 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:28.205 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:28.205 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:28.205 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:28.206 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:28.206 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:28.206 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:28.206 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:28.206 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:28.207 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:28.207 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:28.207 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:28.207 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:28.207 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:28.208 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:28.208 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:28.208 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:28.208 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:28.208 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:28.208 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:28.209 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:28.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:28.209 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:28.209 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:28.209 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:28.210 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:28.210 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:28.210 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:28.210 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:28.210 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:28.210 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:28.211 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:28.211 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:28.211 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:28.211 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:28.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:28.212 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:28.212 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:28.212 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:28.212 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:28.213 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:28.214 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.492727_137.1 2023-10-12T19:23:33.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:33.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:33.217 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:33.268 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:33.269 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:33.329 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:33.330 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:33.330 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:33.330 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:33.331 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:33.331 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:33.331 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:33.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:33.331 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:33.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:33.332 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:33.332 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:33.332 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:33.332 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:23:33.332 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:33.333 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:33.333 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:33.333 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:23:33.333 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:33.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:33.334 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:33.334 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:33.334 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:33.334 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:33.335 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:33.335 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:33.335 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:33.335 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:33.335 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:33.336 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:33.336 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:33.336 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:33.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:33.336 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:33.337 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:33.337 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:33.337 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:33.337 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:33.338 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:33.338 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:33.338 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:33.338 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:33.338 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:33.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:33.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:33.339 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:33.339 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:33.339 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:33.340 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:33.340 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:33.340 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:33.340 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:33.340 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:33.341 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:33.341 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:33.341 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:33.341 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:33.341 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:33.341 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:33.342 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:33.342 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:33.342 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:33.342 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:33.343 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:33.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.410419_126.1 2023-10-12T19:23:38.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:38.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:38.346 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:38.388 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:38.389 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:38.440 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:38.441 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:38.441 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:38.441 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:38.441 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:38.442 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:38.442 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:38.442 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:38.442 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:38.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:38.443 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:38.443 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:38.443 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:38.443 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T19:23:38.443 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:38.443 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:38.444 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:38.444 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T19:23:38.444 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:38.444 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:38.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:38.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:38.445 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:38.445 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:38.445 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:38.445 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:38.446 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:38.446 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:38.446 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:38.446 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:38.446 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:38.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:38.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:38.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:38.447 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:38.447 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:38.447 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:38.448 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:38.448 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:38.448 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:38.448 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:38.448 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:38.449 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:38.449 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:38.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:38.449 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:38.449 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:38.449 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:38.450 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:38.450 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:38.450 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:38.450 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:38.450 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:38.451 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:38.451 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:38.451 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:38.451 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:38.451 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:38.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:38.452 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:38.452 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:38.452 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:38.452 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:38.453 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:38.454 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.438188_130.1 2023-10-12T19:23:43.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:43.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:43.456 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:43.514 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:43.514 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:43.567 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:43.568 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:43.568 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:43.568 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:43.568 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:43.568 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:43.569 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:43.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:43.569 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:43.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:43.569 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:43.570 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:43.570 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:43.570 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470052_132.1", 2023-10-12T19:23:43.570 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:43.570 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:43.571 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:43.571 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470052Z", 2023-10-12T19:23:43.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:43.571 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:43.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:43.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:43.572 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:43.572 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:43.572 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:43.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:43.572 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:43.573 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:43.573 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:43.573 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:43.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:43.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:43.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:43.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:43.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:43.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:43.574 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:43.574 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:43.575 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:43.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:43.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:43.575 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:43.575 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:43.576 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:43.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:43.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:43.576 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:43.576 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:43.576 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:43.577 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:43.577 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:43.577 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:43.577 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:43.578 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:43.578 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:43.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:43.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:43.578 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:43.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:43.579 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:43.579 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:43.579 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:43.579 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:43.580 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:43.581 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470052_132.1 2023-10-12T19:23:48.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:48.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:48.584 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:48.626 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:48.627 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:48.679 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:48.680 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:48.680 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:48.680 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:48.680 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:48.680 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:48.681 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:48.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:48.681 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:48.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:48.681 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:48.682 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:48.682 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:48.682 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:23:48.682 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:48.682 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:48.683 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:48.683 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:23:48.683 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:48.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:48.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:48.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:48.684 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:48.684 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:48.684 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:48.684 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:48.684 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:48.685 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:48.685 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:48.685 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:48.685 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:48.685 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:48.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:48.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:48.686 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:48.686 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:48.686 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:48.686 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:48.687 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:48.687 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:48.687 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:48.687 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:48.687 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:48.688 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:48.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:48.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:48.688 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:48.688 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:48.689 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:48.689 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:48.689 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:48.689 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:48.689 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:48.689 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:48.690 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:48.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:48.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:48.690 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:48.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:48.691 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:48.691 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:48.691 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:48.691 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:48.691 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:48.693 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470944_133.1 2023-10-12T19:23:53.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:53.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:53.695 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:53.791 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:53.791 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:53.877 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:53.878 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:53.878 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:53.878 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:53.878 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:53.879 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:53.879 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:53.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:53.879 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:53.879 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:53.880 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:53.880 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:53.880 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:53.880 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:23:53.880 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:53.881 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:53.881 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:53.881 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:23:53.881 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:53.881 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:53.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:53.882 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:53.882 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:53.882 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:53.882 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:53.883 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:53.883 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:53.883 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:53.883 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:53.883 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:53.884 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:53.884 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:53.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:53.884 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:53.884 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:53.884 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:53.885 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:53.885 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:53.885 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:53.885 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:53.885 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:53.886 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:53.886 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:53.886 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:53.886 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:53.886 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:53.887 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:53.887 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:53.887 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:53.887 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:53.887 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:53.888 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:53.888 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:53.888 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:53.888 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:53.888 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:53.888 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:53.889 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:53.889 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:53.889 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:53.889 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:53.889 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:53.890 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:53.890 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:53.891 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.436071_129.1 2023-10-12T19:23:58.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:58.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:23:58.893 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:23:58.974 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:23:58.974 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:23:59.167 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:23:59.167 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:23:59.167 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:23:59.167 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:23:59.168 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:23:59.168 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:23:59.168 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:59.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:59.168 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:23:59.169 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:59.169 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:23:59.170 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:59.170 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:59.170 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:23:59.171 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:59.171 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:23:59.171 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:59.172 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:23:59.172 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:59.172 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:59.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:59.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:59.173 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:23:59.174 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:59.174 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:59.174 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:23:59.175 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:59.175 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:59.176 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:59.176 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:23:59.176 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:59.176 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:59.177 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:59.177 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:59.177 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:23:59.177 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:59.178 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:59.178 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:59.178 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:59.178 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:59.178 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:59.179 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:59.179 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:59.179 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:59.179 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:59.179 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:23:59.180 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:23:59.180 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:23:59.180 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:23:59.180 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:23:59.180 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:23:59.181 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:23:59.181 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:23:59.181 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:23:59.181 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:23:59.181 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:59.182 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:59.183 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:23:59.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:23:59.184 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:23:59.184 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:23:59.184 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:23:59.184 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:23:59.184 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:23:59.185 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.432502_127.1 2023-10-12T19:24:04.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:04.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:04.188 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:04.234 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:04.234 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:04.293 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:04.293 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:04.294 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:04.294 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:04.294 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:04.294 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:04.295 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:04.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:04.295 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:04.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:04.295 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:04.296 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:04.296 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:04.296 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:24:04.296 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:04.296 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:04.297 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:04.297 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:24:04.297 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:04.297 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:04.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:04.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:04.298 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:04.298 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:04.298 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:04.299 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:04.299 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:04.299 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:04.299 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:04.299 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:04.300 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:04.300 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:04.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:04.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:04.300 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:04.301 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:04.301 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:04.301 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:04.301 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:04.301 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:04.302 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:04.302 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:04.302 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:04.302 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:04.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:04.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:04.303 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:04.303 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:04.303 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:04.304 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:04.304 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:04.304 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:04.304 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:04.305 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:04.305 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:04.305 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:04.305 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:04.305 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:04.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:04.306 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:04.306 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:04.306 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:04.306 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:04.307 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:04.308 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.432502_127.1 2023-10-12T19:24:09.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:09.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:09.310 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:09.353 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:09.353 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:09.404 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:09.405 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:09.405 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:09.405 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:09.405 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:09.405 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:09.406 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:09.406 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:09.406 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:09.406 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:09.406 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:09.407 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:09.407 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:09.407 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:24:09.407 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:09.407 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:09.408 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:09.408 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:24:09.408 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:09.408 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:09.408 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:09.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:09.409 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:09.409 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:09.409 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:09.409 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:09.409 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:09.410 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:09.410 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:09.410 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:09.410 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:09.410 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:09.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:09.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:09.411 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:09.411 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:09.411 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:09.412 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:09.412 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:09.412 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:09.412 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:09.412 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:09.412 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:09.413 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:09.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:09.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:09.413 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:09.413 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:09.414 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:09.414 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:09.414 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:09.414 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:09.415 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:09.415 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:09.415 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:09.415 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:09.415 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:09.416 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:09.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:09.416 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:09.416 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:09.416 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:09.417 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:09.417 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:09.418 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.436071_129.1 2023-10-12T19:24:14.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:14.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:14.420 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:14.469 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:14.469 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:14.529 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:14.529 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:14.529 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:14.530 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:14.530 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:14.530 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:14.530 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:14.530 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:14.531 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:14.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:14.531 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:14.531 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:14.531 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:14.532 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:24:14.532 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:14.532 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:14.532 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:14.532 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:24:14.533 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:14.533 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:14.533 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:14.533 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:14.533 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:14.534 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:14.534 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:14.535 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:14.535 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:14.535 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:14.535 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:14.535 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:14.535 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:14.536 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:14.536 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:14.536 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:14.536 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:14.536 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:14.537 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:14.537 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:14.537 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:14.537 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:14.537 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:14.538 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:14.538 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:14.538 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:14.538 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:14.538 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:14.539 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:14.539 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:14.539 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:14.539 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:14.539 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:14.540 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:14.540 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:14.540 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:14.540 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:14.540 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:14.541 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:14.541 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:14.541 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:14.541 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:14.541 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:14.541 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:14.542 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:14.542 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:14.543 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470944_133.1 2023-10-12T19:24:19.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:19.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:19.545 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:19.645 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:19.646 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:19.722 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:19.723 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:19.723 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:19.723 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:19.724 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:19.724 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:19.724 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:19.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:19.724 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:19.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:19.725 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:19.725 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:19.725 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:19.725 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:24:19.726 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:19.726 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:19.726 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:19.726 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:24:19.726 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:19.727 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:19.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:19.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:19.727 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:19.727 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:19.728 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:19.728 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:19.728 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:19.728 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:19.728 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:19.729 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:19.729 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:19.729 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:19.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:19.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:19.730 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:19.730 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:19.730 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:19.730 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:19.730 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:19.731 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:19.731 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:19.731 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:19.731 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:19.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:19.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:19.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:19.732 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:19.732 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:19.732 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:19.732 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:19.733 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:19.733 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:19.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:19.733 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:19.733 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:19.733 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:19.734 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:19.734 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:19.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:19.734 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:19.734 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:19.735 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:19.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:19.735 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:19.736 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.410419_126.1 2023-10-12T19:24:24.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:24.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:24.739 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:24.851 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:24.851 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:25.081 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:25.081 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:25.081 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:25.082 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:25.082 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:25.082 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:25.082 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:25.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:25.083 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:25.083 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:25.083 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:25.083 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:25.083 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:25.084 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:24:25.084 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:25.084 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:25.085 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:25.085 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:24:25.085 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:25.085 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:25.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:25.085 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:25.086 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:25.086 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:25.086 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:25.086 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:25.086 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:25.087 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:25.087 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:25.087 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:25.087 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:25.088 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:25.088 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:25.088 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:25.088 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:25.088 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:25.089 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:25.089 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:25.089 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:25.089 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:25.089 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:25.090 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:25.090 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:25.090 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:25.090 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:25.090 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:25.091 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:25.091 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:25.091 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:25.091 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:25.091 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:25.092 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:25.092 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:25.092 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:25.093 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:25.093 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:25.093 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:25.093 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:25.093 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:25.093 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:25.094 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:25.094 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:25.094 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:25.095 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:25.095 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.436071_129.1 2023-10-12T19:24:30.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:30.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:30.098 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:30.176 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:30.177 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:30.377 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:30.377 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:30.378 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:30.378 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:30.378 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:30.378 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:30.378 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:30.379 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:30.379 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:30.379 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:30.379 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:30.379 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:30.380 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:30.380 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:24:30.380 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:30.380 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:30.380 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:30.381 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:24:30.381 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:30.381 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:30.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:30.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:30.381 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:30.382 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:30.382 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:30.382 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:30.382 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:30.382 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:30.383 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:30.383 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:30.383 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:30.383 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:30.384 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:30.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:30.384 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:30.384 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:30.384 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:30.385 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:30.385 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:30.385 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:30.385 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:30.385 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:30.385 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:30.386 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:30.386 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:30.386 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:30.386 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:30.387 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:30.387 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:30.387 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:30.387 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:30.387 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:30.388 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:30.388 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:30.388 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:30.388 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:30.388 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:30.389 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:30.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:30.389 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:30.389 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:30.389 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:30.390 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:30.390 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:30.391 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.470944_133.1 2023-10-12T19:24:35.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:35.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:35.394 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:35.475 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:35.476 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:35.563 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:35.563 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:35.564 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:35.564 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:35.564 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:35.564 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:35.564 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:35.565 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:35.565 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:35.565 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:35.565 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:35.565 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:35.566 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:35.566 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:24:35.566 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:35.566 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:35.566 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:35.567 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:24:35.567 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:35.567 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:35.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:35.567 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:35.568 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:35.568 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:35.568 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:35.568 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:35.568 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:35.569 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:35.569 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:35.569 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:35.570 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:35.570 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:35.570 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:35.570 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:35.571 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:35.571 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:35.571 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:35.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:35.572 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:35.572 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:35.573 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:35.573 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:35.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:35.574 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:35.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:35.576 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:35.576 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:35.576 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:35.576 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:35.577 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:35.577 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:35.577 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:35.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:35.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:35.578 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:35.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:35.579 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:35.580 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:35.580 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:35.580 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:35.581 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:35.582 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.410419_126.1 2023-10-12T19:24:40.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:40.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:40.584 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:40.663 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:40.663 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:41.066 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:24:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:24:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:41.077 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:41.077 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:41.077 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:41.077 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:41.078 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:41.078 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:41.078 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:41.078 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:41.080 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:41.080 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:41.081 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138371.171804_293.1 target=1_1697136751.436071_129.1 2023-10-12T19:24:46.083 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2023-10-12T19:24:46.084 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-48 2023-10-12T19:24:46.110 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-49 2023-10-12T19:24:46.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:46.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:46.287 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:46.332 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:46.333 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:46.373 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T19:24:46.385 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T19:24:46.385 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:46.386 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697138686.129941_308.1", 2023-10-12T19:24:46.386 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T19:24:46.129941Z" 2023-10-12T19:24:46.386 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:46.386 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:46.386 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:46.387 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T19:24:46.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:46.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:46.387 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:46.387 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:24:46.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:46.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:46.388 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:46.388 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:24:46.388 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:46.389 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T19:24:46.390 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T19:24:46.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:46.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:46.391 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:46.435 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:46.436 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:46.790 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:46.790 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:46.790 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:46.791 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:46.791 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:46.791 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:46.791 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:46.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:46.792 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:46.792 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:46.792 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:46.792 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:46.792 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:46.793 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:24:46.793 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:46.793 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:46.793 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:46.793 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:24:46.794 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:46.794 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:46.794 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:46.794 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:46.794 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:46.795 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:46.795 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:46.795 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:46.795 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:46.795 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:46.796 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:46.796 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:46.796 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:46.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:46.797 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:46.797 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:46.797 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:46.797 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:46.798 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:46.798 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:46.798 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:46.798 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:46.799 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:46.799 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:46.799 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:46.799 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:46.799 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:46.800 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:46.800 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:46.800 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:46.800 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:46.800 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:46.800 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:46.801 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:46.801 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:46.801 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:46.801 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:46.801 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:46.802 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:46.802 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:46.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:46.802 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:46.802 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:46.803 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:46.803 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:46.803 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:46.803 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:24:51.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:51.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:51.806 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:51.850 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:51.850 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:51.903 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:51.903 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:51.903 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:51.903 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:51.904 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:51.904 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:51.904 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:51.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:51.905 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:51.905 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:51.905 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:51.905 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:51.905 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:51.906 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:24:51.906 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:51.906 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:51.906 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:51.907 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:24:51.907 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:51.907 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:51.907 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:51.907 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:51.908 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:51.908 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:51.908 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:51.908 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:51.909 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:51.910 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:51.910 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:51.910 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:51.910 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:51.911 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:51.911 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:51.911 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:51.911 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:51.911 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:51.912 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:51.912 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:51.912 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:51.912 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:51.912 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:51.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:51.913 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:51.913 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:51.913 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:51.914 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:51.915 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:51.916 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:51.917 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:51.918 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:24:56.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:56.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:24:56.920 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:24:56.963 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:24:56.963 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:24:57.015 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:24:57.016 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:24:57.016 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:24:57.016 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:24:57.016 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:24:57.018 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:24:57.018 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:57.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:57.019 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:24:57.019 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:57.019 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:24:57.019 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:57.019 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:57.020 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.492727_137.1", 2023-10-12T19:24:57.020 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:57.020 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:24:57.020 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:57.020 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.492727Z", 2023-10-12T19:24:57.021 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:57.021 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:57.021 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:57.021 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:57.021 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:24:57.022 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:57.022 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:57.022 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:24:57.022 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:57.022 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:57.023 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:57.023 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:24:57.023 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:57.023 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:57.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:57.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:57.024 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:24:57.024 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:57.024 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:57.024 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:57.025 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:57.025 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:57.025 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:57.025 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:57.026 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:57.026 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:57.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:57.026 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:24:57.027 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:24:57.027 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:24:57.027 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:24:57.027 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:24:57.027 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:24:57.028 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:24:57.028 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:24:57.028 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:24:57.028 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:24:57.028 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:24:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:24:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:24:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:24:57.029 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:24:57.030 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:24:57.030 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:24:57.031 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.492727_137.1 2023-10-12T19:25:02.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:02.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:02.033 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:02.076 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:02.077 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:02.129 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:02.129 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:02.130 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:02.130 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:02.130 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:02.130 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:02.131 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:02.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:02.131 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:02.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:02.132 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:02.132 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:02.132 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:02.132 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:25:02.133 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:02.133 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:02.133 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:02.133 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:25:02.133 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:02.133 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:02.134 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:02.134 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:02.134 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:02.134 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:02.134 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:02.135 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:02.135 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:02.135 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:02.135 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:02.135 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:02.136 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:02.136 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:02.136 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:02.136 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:02.136 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:02.137 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:02.137 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:02.137 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:02.138 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:02.138 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:02.138 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:02.138 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:02.138 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:02.139 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:02.139 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:02.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:02.139 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:02.139 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:02.140 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:02.140 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:02.140 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:02.140 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:02.140 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:02.141 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:02.141 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:02.141 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:02.142 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:02.142 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:02.142 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:02.142 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:02.143 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:02.143 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:02.143 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:02.143 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:02.144 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.432502_127.1 2023-10-12T19:25:07.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:07.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:07.147 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:07.191 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:07.191 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:07.283 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:07.283 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:07.283 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:07.284 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:07.284 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:07.284 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:07.284 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:07.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:25:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:25:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:07.296 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:07.297 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.407338_124.1 2023-10-12T19:25:12.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:12.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:12.301 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:12.402 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:12.402 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:12.620 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:12.621 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:12.621 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:12.621 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:12.621 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:12.622 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:12.622 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:12.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:12.622 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:12.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:12.623 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:12.623 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:12.623 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:12.623 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:25:12.623 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:12.624 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:12.624 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:12.624 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:25:12.624 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:12.624 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:12.625 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:12.625 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:12.625 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:12.625 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:12.625 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:12.625 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:12.626 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:12.626 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:12.626 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:12.626 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:12.626 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:12.627 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:12.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:12.627 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:12.627 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:12.628 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:12.628 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:12.628 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:12.628 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:12.628 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:12.628 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:12.629 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:12.629 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:12.630 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:12.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:12.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:12.630 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:12.631 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:12.631 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:12.631 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:12.631 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:12.631 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:12.632 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:12.632 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:12.632 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:12.632 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:12.632 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:12.632 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:12.633 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:12.633 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:12.633 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:12.633 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:12.633 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:12.634 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:12.635 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:25:17.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:17.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:17.638 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:17.712 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:17.713 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:18.143 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:18.144 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:18.144 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:18.144 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:18.144 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:18.145 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:18.145 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:18.145 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:18.145 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:18.146 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:25:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:25:18.147 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:18.148 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:18.149 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:18.150 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:18.150 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:18.151 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:18.151 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:18.151 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:18.151 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:18.152 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:18.153 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:18.153 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:18.153 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:18.153 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:18.153 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:18.154 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:18.154 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:18.154 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:18.154 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:18.154 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:18.155 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:18.155 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:18.155 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:18.155 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:18.155 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:18.156 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:18.156 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:18.156 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:18.156 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:18.156 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:18.156 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:18.157 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:18.157 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:18.158 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.432502_127.1 2023-10-12T19:25:23.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:23.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:23.161 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:23.237 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:23.237 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:23.671 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:23.671 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:23.671 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:23.671 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:23.671 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:23.672 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:23.672 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:23.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:23.672 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:23.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:23.673 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:23.673 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:23.673 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:23.673 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:25:23.673 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:23.674 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:23.674 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:23.674 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:25:23.674 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:23.674 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:23.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:23.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:23.675 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:23.675 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:23.675 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:23.676 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:23.676 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:23.676 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:23.676 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:23.676 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:23.676 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:23.677 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:23.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:23.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:23.677 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:23.678 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:23.678 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:23.678 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:23.678 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:23.678 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:23.679 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:23.679 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:23.679 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:23.679 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:23.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:23.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:23.680 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:23.680 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:23.680 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:23.680 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:23.680 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:23.681 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:23.681 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:23.681 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:23.681 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:23.681 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:23.682 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:23.682 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:23.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:23.682 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:23.682 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:23.683 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:23.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:23.683 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:23.683 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.432502_127.1 2023-10-12T19:25:28.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:28.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:28.686 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:28.728 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:28.728 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:28.784 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:28.785 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:28.785 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:28.785 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:28.785 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:28.785 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:28.786 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:28.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:28.786 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:28.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:28.786 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:28.787 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:28.787 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:28.787 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:25:28.787 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:28.787 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:28.788 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:28.788 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:25:28.788 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:28.788 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:28.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:28.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:28.789 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:28.789 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:28.789 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:28.789 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:28.789 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:28.790 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:28.790 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:28.790 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:28.790 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:28.790 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:28.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:28.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:28.791 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:28.791 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:28.791 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:28.792 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:28.792 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:28.792 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:28.792 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:28.792 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:28.792 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:28.793 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:28.793 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:28.793 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:28.793 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:28.793 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:28.794 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:28.794 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:28.794 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:28.794 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:28.794 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:28.795 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:28.795 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:28.795 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:28.795 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:28.795 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:28.795 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:28.796 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:28.796 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:28.796 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:28.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:28.796 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:28.798 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:25:33.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:33.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:33.800 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:33.841 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:33.842 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:33.893 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:33.893 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:33.893 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:33.894 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:33.894 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:33.894 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:33.895 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:33.895 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:33.896 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:33.896 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:33.896 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:33.897 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:33.897 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:33.897 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:25:33.897 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:33.898 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:33.898 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:33.898 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:25:33.898 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:33.898 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:33.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:33.899 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:33.899 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:33.899 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:33.899 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:33.899 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:33.900 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:33.900 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:33.900 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:33.900 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:33.900 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:33.901 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:33.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:33.901 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:33.901 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:33.901 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:33.902 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:33.902 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:33.902 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:33.902 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:33.902 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:33.903 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:33.903 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:33.903 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:33.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:33.903 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:33.904 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:33.904 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:33.904 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:33.904 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:33.904 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:33.905 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:33.905 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:33.905 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:33.905 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:33.905 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:33.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:33.906 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:33.906 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:33.907 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:33.907 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:33.907 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:33.907 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:33.908 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:33.909 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:25:38.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:38.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:38.911 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:38.969 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:38.969 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:39.022 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:39.022 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:39.023 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:39.023 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:39.023 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:39.023 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:39.023 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:39.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:39.024 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:39.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:39.024 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:39.024 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:39.025 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:39.025 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:25:39.025 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:39.025 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:39.025 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:39.026 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:25:39.026 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:39.026 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:39.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:39.027 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:39.027 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:39.027 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:39.027 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:39.027 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:39.028 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:39.028 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:39.028 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:39.028 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:39.028 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:39.029 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:39.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:39.029 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:39.029 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:39.029 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:39.030 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:39.030 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:39.030 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:39.030 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:39.030 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:39.031 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:39.031 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:39.031 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:39.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:39.031 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:39.032 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:39.032 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:39.032 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:39.032 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:39.032 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:39.033 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:39.033 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:39.033 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:39.033 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:39.033 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:39.034 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:39.034 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:39.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:39.034 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:39.034 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:39.035 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:39.035 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:39.035 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:39.036 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:25:44.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:44.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:44.039 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:44.102 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:44.102 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:44.162 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:44.162 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:44.162 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:44.163 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:44.163 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:44.163 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:44.163 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:44.163 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:44.164 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:44.164 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:44.164 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:44.164 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:44.164 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:44.165 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:25:44.165 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:44.165 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:44.165 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:44.165 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:25:44.166 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:44.166 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:44.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:44.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:44.166 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:44.167 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:44.167 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:44.167 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:44.167 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:44.168 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:44.168 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:44.168 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:44.168 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:44.168 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:44.169 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:44.169 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:44.169 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:44.169 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:44.169 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:44.169 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:44.170 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:44.170 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:44.170 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:44.170 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:44.170 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:44.171 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:44.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:44.171 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:44.171 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:44.171 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:44.171 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:44.172 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:44.172 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:44.172 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:44.172 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:44.172 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:44.173 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:44.173 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:44.173 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:44.173 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:44.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:44.174 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:44.174 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:44.174 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:44.174 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:44.174 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:44.176 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.436071_129.1 2023-10-12T19:25:49.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:49.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:49.178 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:49.272 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:49.272 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:49.648 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:49.648 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:49.648 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:49.648 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:49.650 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:49.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:49.650 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:49.650 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:49.650 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:49.651 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:25:49.651 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:49.651 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:49.651 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:49.651 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:25:49.652 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:49.652 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:49.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:49.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:49.652 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:49.653 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:49.653 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:49.653 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:49.653 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:49.654 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:49.654 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:49.654 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:49.654 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:49.655 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:49.656 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:49.657 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:49.658 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:49.659 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:49.660 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:49.661 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:49.662 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.436071_129.1 2023-10-12T19:25:54.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:54.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:54.664 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:25:54.750 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:25:54.750 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:25:54.979 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:25:54.980 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:25:54.980 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:25:54.980 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:25:54.980 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:25:54.980 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:25:54.981 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:54.981 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:54.981 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:25:54.981 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:54.981 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:25:54.982 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:54.982 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:54.982 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:25:54.982 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:54.982 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:25:54.983 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:54.983 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:25:54.983 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:54.983 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:54.983 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:54.984 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:54.984 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:25:54.984 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:54.984 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:54.984 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:25:54.985 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:54.985 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:54.985 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:54.985 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:25:54.985 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:54.986 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:54.986 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:54.986 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:54.986 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:25:54.986 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:54.987 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:54.987 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:54.987 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:54.987 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:54.987 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:54.988 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:54.988 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:54.988 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:54.988 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:54.988 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:25:54.988 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:25:54.989 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:25:54.989 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:25:54.989 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:25:54.989 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:25:54.989 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:25:54.990 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:25:54.990 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:25:54.990 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:25:54.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:54.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:54.991 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:25:54.991 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:25:54.991 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:25:54.991 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:25:54.991 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:25:54.992 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:25:54.992 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:25:54.993 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.470944_133.1 2023-10-12T19:25:59.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:59.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:25:59.996 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:00.038 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:00.039 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:00.443 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:00.443 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:00.443 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:00.443 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:00.444 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:00.444 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:00.444 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:00.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:00.444 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:00.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:00.445 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:00.445 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:00.445 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:00.445 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:26:00.445 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:00.446 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:00.446 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:00.446 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:26:00.446 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:00.446 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:00.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:00.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:00.447 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:00.447 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:00.447 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:00.447 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:00.448 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:00.448 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:00.448 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:00.448 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:00.448 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:00.449 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:00.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:00.449 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:00.449 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:00.449 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:00.449 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:00.450 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:00.450 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:00.450 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:00.450 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:00.450 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:00.451 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:00.451 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:00.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:00.451 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:00.451 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:00.451 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:00.452 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:00.452 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:00.452 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:00.452 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:00.452 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:00.453 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:00.453 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:00.453 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:00.453 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:00.453 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:00.453 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:00.454 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:00.454 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:00.454 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:00.454 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:00.454 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:00.455 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.470944_133.1 2023-10-12T19:26:05.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:05.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:05.458 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:05.503 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:05.503 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:05.559 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:05.559 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:05.560 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:05.560 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:05.560 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:05.560 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:05.561 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:05.561 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:05.561 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:05.561 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:05.561 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:05.562 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:05.562 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:05.562 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:26:05.562 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:05.562 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:05.563 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:05.563 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:26:05.563 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:05.563 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:05.563 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:05.563 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:05.564 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:05.564 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:05.564 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:05.564 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:05.564 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:05.565 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:05.565 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:05.565 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:05.565 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:05.565 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:05.566 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:05.566 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:05.566 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:05.566 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:05.566 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:05.567 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:05.567 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:05.567 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:05.567 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:05.567 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:05.567 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:05.568 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:05.568 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:05.568 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:05.568 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:05.568 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:05.569 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:05.569 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:05.569 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:05.569 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:05.569 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:05.570 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:05.570 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:05.570 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:05.570 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:05.570 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:05.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:05.571 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:05.571 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:05.571 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:05.571 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:05.572 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:05.573 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:26:10.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:10.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:10.576 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:10.619 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:10.619 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:10.671 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:10.671 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:10.673 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:10.673 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:10.673 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:10.673 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:10.674 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:10.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:10.674 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:10.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:10.674 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:10.675 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:10.675 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:10.675 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:26:10.675 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:10.675 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:10.676 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:10.676 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:26:10.676 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:10.676 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:10.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:10.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:10.677 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:10.677 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:10.677 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:10.677 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:10.677 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:10.678 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:10.678 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:10.678 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:10.678 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:10.678 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:10.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:10.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:10.679 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:10.679 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:10.679 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:10.680 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:10.680 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:10.680 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:10.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:10.680 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:10.681 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:10.681 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:10.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:10.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:10.681 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:10.681 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:10.682 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:10.682 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:10.682 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:10.682 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:10.682 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:10.683 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:10.683 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:10.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:10.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:10.683 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:10.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:10.684 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:10.684 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:10.684 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:10.684 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:10.684 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:10.686 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.407338_124.1 2023-10-12T19:26:15.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:15.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:15.689 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:15.776 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:15.776 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:15.927 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:15.928 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:15.928 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:15.928 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:15.928 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:15.929 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:15.929 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:15.929 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:15.929 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:15.929 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:15.930 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:15.930 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:15.930 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:15.930 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:26:15.930 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:15.931 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:15.931 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:15.931 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:26:15.931 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:15.931 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:15.932 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:15.933 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:15.933 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:15.933 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:15.933 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:15.933 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:15.934 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:15.934 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:15.934 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:15.937 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:15.937 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:15.937 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:15.937 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:15.938 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:15.938 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:15.938 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:15.938 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:15.938 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:15.939 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:15.939 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:15.939 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:15.939 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:15.939 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:15.939 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:15.940 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:15.940 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:15.940 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:15.940 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:15.940 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:15.941 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:15.941 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:15.941 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:15.941 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:15.941 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:15.942 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:15.942 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:15.942 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:15.942 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:15.942 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:15.943 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.407338_124.1 2023-10-12T19:26:20.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:20.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:20.946 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:20.988 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:20.989 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:21.041 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:21.042 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:21.042 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:21.042 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:21.042 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:21.043 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:21.043 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:21.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:21.043 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:21.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:21.044 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:21.044 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:21.044 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:21.044 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:26:21.044 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:21.045 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:21.045 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:21.045 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:26:21.045 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:21.045 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:21.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:21.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:21.046 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:21.046 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:21.046 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:21.046 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:21.047 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:21.047 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:21.047 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:21.047 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:21.047 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:21.048 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:21.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:21.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:21.048 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:21.048 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:21.049 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:21.049 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:21.049 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:21.049 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:21.049 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:21.050 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:21.050 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:21.050 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:21.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:21.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:21.050 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:21.051 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:21.051 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:21.051 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:21.051 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:21.051 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:21.053 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:21.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:21.053 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:21.053 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:21.053 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:21.054 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:21.054 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:21.055 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:26:26.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:26.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:26.058 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:26.130 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:26.131 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:26.594 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:26.594 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:26.595 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:26.595 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:26.595 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:26.595 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:26.595 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:26.596 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:26.596 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:26.596 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:26.596 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:26.596 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:26.597 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:26.597 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:26:26.597 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:26.597 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:26.598 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:26.598 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:26:26.598 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:26.598 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:26.599 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:26.599 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:26.599 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:26.599 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:26.599 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:26.600 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:26.600 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:26.600 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:26.600 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:26.600 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:26.600 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:26.601 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:26.601 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:26.601 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:26.601 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:26.602 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:26.602 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:26.602 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:26.602 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:26.602 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:26.603 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:26.603 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:26.603 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:26.603 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:26.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:26.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:26.604 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:26.604 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:26.604 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:26.605 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:26.605 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:26.605 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:26.605 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:26.605 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:26.606 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:26.606 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:26.606 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:26.606 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:26.606 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:26.607 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:26.607 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:26.607 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:26.607 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:26.608 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:26.609 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:26:31.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:31.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:31.611 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:31.667 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:31.667 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:32.165 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:32.166 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:32.166 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:32.167 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:32.167 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:32.167 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:32.168 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:32.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:32.168 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:32.169 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:32.169 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:32.170 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:32.170 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:32.171 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:26:32.171 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:32.171 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:32.172 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:32.172 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:26:32.172 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:32.173 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:32.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:32.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:32.174 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:32.175 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:32.175 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:32.175 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:32.176 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:32.176 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:32.176 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:32.177 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:32.177 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:32.177 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:32.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:32.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:32.179 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:32.179 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:32.179 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:32.180 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:32.180 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:32.180 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:32.180 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:32.181 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:32.181 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:32.181 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:32.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:32.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:32.182 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:32.182 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:32.182 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:32.182 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:32.183 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:32.183 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:32.183 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:32.183 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:32.185 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:32.185 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:32.185 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:32.185 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:32.185 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:32.186 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:32.186 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:32.186 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:32.186 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:32.186 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:32.188 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.470944_133.1 2023-10-12T19:26:37.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:37.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:37.190 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:37.262 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:37.262 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:37.682 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:37.683 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:37.683 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:37.684 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:37.684 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:37.684 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:37.685 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:37.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:37.685 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:37.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:37.685 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:37.686 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:37.686 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:37.686 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.545187_139.1", 2023-10-12T19:26:37.686 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:37.687 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:37.687 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:37.687 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.545187Z", 2023-10-12T19:26:37.687 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:37.687 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:37.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:37.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:37.688 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:37.688 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:37.688 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:37.688 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:37.689 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:37.689 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:37.689 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:37.689 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:37.690 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:37.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:37.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:37.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:37.691 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:37.691 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:37.691 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:37.691 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:37.691 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:37.692 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:37.692 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:37.692 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:37.692 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:37.692 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:37.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:37.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:37.693 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:37.693 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:37.693 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:37.693 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:37.694 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:37.694 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:37.694 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:37.694 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:37.695 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:37.695 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:37.695 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:37.695 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:37.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:37.696 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:37.696 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:37.696 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:37.696 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:37.696 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:37.697 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.545187_139.1 2023-10-12T19:26:42.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:42.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:42.700 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:42.743 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:42.744 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:42.799 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:42.799 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:42.799 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:42.799 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:42.800 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:42.800 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:42.800 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:42.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:42.801 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:42.801 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:42.801 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:42.801 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:42.801 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:42.802 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:26:42.802 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:42.802 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:42.802 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:42.802 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:26:42.803 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:42.803 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:42.803 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:42.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:42.803 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:42.804 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:42.804 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:42.804 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:42.804 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:42.804 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:42.805 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:42.805 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:42.805 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:42.805 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:42.805 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:42.805 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:42.806 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:42.806 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:42.806 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:42.806 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:42.806 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:42.807 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:42.807 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:42.807 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:42.807 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:42.807 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:42.808 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:42.808 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:42.808 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:42.808 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:42.808 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:42.809 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:42.809 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:42.809 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:42.809 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:42.809 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:42.810 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:42.810 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:42.810 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:42.810 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:42.810 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:42.811 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:42.811 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:42.811 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:42.811 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:42.812 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:42.813 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:26:47.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:47.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:47.815 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:47.858 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:47.858 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:47.909 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:47.909 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:47.910 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:47.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:47.911 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:47.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:47.911 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:47.911 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:47.912 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:47.912 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:26:47.912 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:47.912 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:47.912 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:47.913 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:26:47.913 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:47.913 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:47.913 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:47.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:47.914 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:47.914 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:47.914 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:47.914 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:47.914 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:47.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:47.916 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:47.916 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:47.916 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:47.916 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:47.916 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:47.918 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:47.918 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:47.918 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:47.918 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:47.918 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:47.919 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:47.919 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:47.919 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:47.919 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:47.919 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:47.920 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:47.920 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:47.920 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:47.920 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:47.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:47.921 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:47.921 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:47.921 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:47.921 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:47.921 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:47.922 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.407338_124.1 2023-10-12T19:26:52.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:52.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:52.925 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:53.007 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:53.007 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:53.124 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:53.124 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:53.124 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:53.125 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:53.125 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:53.125 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:53.125 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:53.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:53.126 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:53.126 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:53.126 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:53.126 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:53.126 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:53.127 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:26:53.127 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:53.127 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:53.127 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:53.127 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:26:53.127 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:53.128 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:53.128 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:53.128 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:53.128 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:53.128 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:53.129 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:53.129 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:53.129 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:53.129 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:53.129 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:53.130 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:53.130 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:53.130 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:53.130 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:53.130 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:53.131 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:53.131 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:53.131 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:53.131 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:53.131 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:53.132 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:53.132 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:53.132 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:53.132 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:53.132 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:53.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:53.133 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:53.133 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:53.133 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:53.133 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:53.134 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:53.134 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:53.134 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:53.134 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:53.134 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:53.134 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:53.135 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:53.135 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:53.135 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:53.135 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:53.136 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:53.136 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:53.136 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:53.136 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:53.136 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:53.137 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:26:58.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:58.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:26:58.140 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:26:58.182 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:26:58.182 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:26:58.434 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:26:58.435 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:26:58.435 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:26:58.435 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:26:58.435 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:26:58.436 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:26:58.436 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:58.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:58.436 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:26:58.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:26:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:26:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:58.438 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:26:58.438 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:58.438 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:26:58.438 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:58.439 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:58.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:58.439 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:58.439 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:26:58.439 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:58.439 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:58.440 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:26:58.440 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:58.440 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:58.440 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:26:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:26:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:26:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:26:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:26:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:26:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:26:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:26:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:26:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:26:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:26:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:26:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:26:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:26:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:26:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:26:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:26:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:26:58.447 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:26:58.447 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:27:03.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:03.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:03.450 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:03.549 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:03.549 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:03.828 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:03.828 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:03.828 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:03.829 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:03.829 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:03.829 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:03.829 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:03.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:03.830 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:03.830 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:03.830 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:03.830 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:03.831 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:03.831 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:27:03.831 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:03.831 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:03.831 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:03.831 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:27:03.832 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:03.832 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:03.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:03.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:03.832 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:03.833 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:03.833 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:03.833 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:03.833 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:03.833 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:03.834 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:03.834 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:03.834 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:03.834 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:03.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:03.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:03.835 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:03.835 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:03.835 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:03.835 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:03.835 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:03.836 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:03.836 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:03.836 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:03.836 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:03.836 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:03.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:03.837 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:03.837 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:03.837 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:03.837 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:03.838 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:03.838 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:03.838 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:03.838 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:03.838 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:03.838 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:03.839 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:03.839 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:03.839 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:03.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:03.839 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:03.840 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:03.840 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:03.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:03.840 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:03.841 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.407338_124.1 2023-10-12T19:27:08.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:08.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:08.844 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:08.888 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:08.889 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:08.976 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:08.976 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:08.977 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:08.977 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:08.977 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:08.977 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:08.977 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:08.978 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:08.978 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:08.978 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:08.978 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:08.978 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:08.979 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:08.979 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:27:08.979 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:08.979 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:08.980 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:08.980 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:27:08.980 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:08.980 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:08.981 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:08.981 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:08.981 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:08.981 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:08.981 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:08.982 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:08.982 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:08.982 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:08.982 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:08.982 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:08.983 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:08.983 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:08.983 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:08.983 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:08.983 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:08.984 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:08.984 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:08.985 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:08.985 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:08.985 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:08.986 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:08.986 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:08.986 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:08.987 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:08.987 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:08.988 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:08.988 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:08.988 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:08.989 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:08.989 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:08.989 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:08.989 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:08.989 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:08.990 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:08.990 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:08.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:08.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:08.990 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:08.991 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:08.991 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:08.991 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:08.991 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:08.991 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:08.992 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:08.993 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.436071_129.1 2023-10-12T19:27:13.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:13.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:13.995 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:14.038 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:14.038 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:14.099 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:14.099 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:14.100 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:14.100 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:14.100 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:14.100 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:14.100 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:14.101 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:14.101 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:14.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:14.101 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:14.101 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:14.102 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:14.102 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.488676_136.1", 2023-10-12T19:27:14.102 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:14.102 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:14.102 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:14.103 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.488676Z", 2023-10-12T19:27:14.103 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:14.103 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:14.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:14.103 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:14.104 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:14.104 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:14.104 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:14.104 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:14.104 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:14.105 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:14.105 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:14.105 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:14.105 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:14.105 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:14.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:14.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:14.106 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:14.106 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:14.106 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:14.107 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:14.107 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:14.107 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:14.107 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:14.107 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:14.108 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:14.108 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:14.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:14.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:14.108 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:14.109 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:14.109 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:14.109 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:14.109 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:14.109 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:14.110 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:14.110 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:14.110 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:14.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:14.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:14.111 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:14.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:14.111 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:14.111 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:14.111 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:14.112 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:14.112 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:14.112 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.488676_136.1 2023-10-12T19:27:19.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:19.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:19.115 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:19.156 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:19.157 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:19.209 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:19.209 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:19.209 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:19.210 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:19.210 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:19.210 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:19.210 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:19.210 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:19.211 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:19.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:19.211 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:19.211 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:19.211 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:19.212 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:27:19.212 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:19.212 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:19.212 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:19.212 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:27:19.213 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:19.213 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:19.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:19.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:19.213 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:19.213 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:19.214 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:19.214 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:19.214 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:19.214 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:19.214 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:19.215 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:19.215 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:19.215 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:19.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:19.215 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:19.216 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:19.216 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:19.216 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:19.216 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:19.216 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:19.216 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:19.217 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:19.217 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:19.217 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:19.217 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:19.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:19.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:19.218 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:19.218 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:19.218 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:19.218 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:19.219 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:19.219 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:19.219 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:19.219 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:19.219 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:19.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:19.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:19.220 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:19.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:19.220 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:19.221 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:19.221 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:19.221 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:19.221 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:19.222 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.435183_128.1 2023-10-12T19:27:24.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:24.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:24.224 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:24.268 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:24.268 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:24.323 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:24.324 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:24.324 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:24.324 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:24.324 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:24.325 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:24.325 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:24.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:24.325 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:24.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:24.326 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:24.326 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:24.326 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:24.326 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.495204_138.1", 2023-10-12T19:27:24.326 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:24.327 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:24.327 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:24.327 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.495204Z", 2023-10-12T19:27:24.327 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:24.327 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:24.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:24.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:24.328 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:24.328 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:24.328 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:24.329 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:24.329 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:24.329 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:24.329 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:24.329 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:24.330 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:24.330 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:24.330 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:24.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:24.331 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:24.331 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:24.331 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:24.331 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:24.331 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:24.332 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:24.332 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:24.332 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:24.332 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:24.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:24.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:24.333 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:24.333 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:24.333 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:24.333 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:24.333 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:24.334 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:24.334 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:24.334 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:24.334 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:24.334 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:24.335 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:24.335 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:24.335 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:24.335 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:24.335 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:24.336 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:24.336 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:24.336 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:24.336 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:24.337 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.495204_138.1 2023-10-12T19:27:29.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:29.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:29.340 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:29.383 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:29.383 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:29.436 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:29.436 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:29.436 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:29.437 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:29.437 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:29.437 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:29.437 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:29.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:29.438 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:29.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:29.438 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:29.438 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:29.439 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:29.439 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:27:29.439 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:29.439 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:29.440 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:29.440 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:27:29.440 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:29.440 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:29.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:29.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:29.441 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:29.441 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:29.442 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:29.442 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:29.442 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:29.442 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:29.442 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:29.443 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:29.443 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:29.443 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:29.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:29.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:29.444 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:29.444 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:29.444 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:29.444 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:29.444 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:29.445 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:29.445 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:29.445 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:29.445 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:29.446 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:29.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:29.446 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:29.446 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:29.446 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:29.447 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:29.447 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:29.447 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:29.447 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:29.447 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:29.448 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:29.448 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:29.448 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:29.448 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:29.448 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:29.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:29.449 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:29.449 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:29.449 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:29.450 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:29.450 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:29.451 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:27:34.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:34.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:34.454 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:34.496 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:34.496 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:34.547 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:34.547 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:34.548 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:34.548 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:34.548 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:34.548 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:34.548 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:34.549 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:34.549 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:34.549 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:34.549 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:34.549 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:34.550 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:34.550 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:27:34.550 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:34.550 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:34.550 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:34.551 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:27:34.551 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:34.551 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:34.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:34.551 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:34.552 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:34.552 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:34.552 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:34.552 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:34.552 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:34.553 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:34.553 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:34.553 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:34.553 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:34.553 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:34.554 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:34.554 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:34.554 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:34.554 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:34.554 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:34.555 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:34.555 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:34.555 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:34.555 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:34.555 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:34.556 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:34.556 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:34.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:34.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:34.556 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:34.557 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:34.557 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:34.557 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:34.557 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:34.557 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:34.557 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:34.558 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:34.558 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:34.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:34.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:34.561 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:34.562 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:34.562 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:34.562 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:34.562 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:34.562 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:34.562 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:34.564 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.432502_127.1 2023-10-12T19:27:39.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:39.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:39.566 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:39.620 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:39.620 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:39.671 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:39.672 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:39.672 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:39.672 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:39.672 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:39.673 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:39.673 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:39.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:39.673 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:39.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:39.674 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:39.674 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:39.674 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:39.674 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:27:39.674 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:39.675 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:39.675 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:39.675 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:27:39.675 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:39.675 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:39.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:39.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:39.676 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:39.676 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:39.676 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:39.677 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:39.677 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:39.677 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:39.677 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:39.677 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:39.678 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:39.678 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:39.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:39.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:39.678 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:39.679 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:39.679 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:39.679 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:39.679 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:39.679 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:39.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:39.680 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:39.680 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:39.680 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:39.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:39.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:39.681 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:39.681 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:39.681 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:39.681 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:39.681 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:39.682 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:39.682 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:39.682 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:39.682 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:39.682 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:39.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:39.683 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:39.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:39.683 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:39.683 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:39.683 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:39.684 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:39.684 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:39.685 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:27:44.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:44.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:44.688 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:44.784 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:44.784 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:44.951 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:44.951 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:44.951 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:44.952 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:44.952 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:44.952 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:44.953 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:44.953 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:44.953 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:44.954 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:44.954 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:44.955 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:44.955 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:44.955 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:27:44.956 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:44.956 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:44.956 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:44.956 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:27:44.957 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:44.957 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:44.957 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:44.957 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:44.958 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:44.958 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:44.959 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:44.959 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:44.959 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:44.959 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:44.959 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:44.960 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:44.960 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:44.960 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:44.960 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:44.960 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:44.960 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:44.961 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:44.961 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:44.961 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:44.961 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:44.961 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:44.962 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:44.962 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:44.962 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:44.962 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:44.962 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:44.963 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:44.963 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:44.963 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:44.963 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:44.963 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:44.963 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:44.964 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:44.964 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:44.964 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:44.964 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:44.964 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:44.965 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:44.965 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:44.965 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:44.965 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:44.965 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:44.966 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:44.966 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:44.967 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:44.968 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.436071_129.1 2023-10-12T19:27:49.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:49.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:49.971 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:50.067 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:50.068 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:50.371 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:50.371 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:50.371 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:50.372 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:50.372 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:50.372 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:50.372 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:50.372 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:50.373 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:50.373 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:50.373 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:50.373 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:50.373 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:50.374 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:27:50.374 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:50.374 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:50.374 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:50.375 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:27:50.375 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:50.375 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:50.375 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:50.375 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:50.376 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:50.376 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:50.376 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:50.376 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:50.376 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:50.377 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:50.377 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:50.377 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:50.377 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:50.377 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:50.378 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:50.378 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:50.378 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:50.378 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:50.379 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:50.379 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:50.379 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:50.379 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:50.379 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:50.380 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:50.380 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:50.380 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:50.380 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:50.380 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:50.381 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:50.381 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:50.381 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:50.381 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:50.381 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:50.382 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:50.382 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:50.382 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:50.382 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:50.382 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:50.383 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:50.383 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:50.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:50.383 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:50.383 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:50.384 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:50.384 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:50.384 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:50.385 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:27:55.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:55.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:27:55.387 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:27:55.458 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:27:55.458 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:27:55.523 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:27:55.524 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:27:55.524 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:27:55.524 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:27:55.524 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:27:55.525 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:27:55.525 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:55.525 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:55.525 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:27:55.525 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:55.526 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:27:55.526 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:55.526 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:55.526 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.488676_136.1", 2023-10-12T19:27:55.526 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:55.527 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:27:55.527 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:55.527 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.488676Z", 2023-10-12T19:27:55.527 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:55.527 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:55.528 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:55.528 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:55.528 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:27:55.528 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:55.528 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:55.529 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:27:55.529 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:55.529 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:55.529 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:55.529 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:27:55.530 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:55.530 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:55.530 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:55.530 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:55.530 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:27:55.531 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:55.531 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:55.531 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:55.531 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:55.531 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:55.532 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:55.532 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:55.532 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:55.532 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:55.532 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:55.532 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:27:55.533 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:27:55.533 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:27:55.533 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:27:55.533 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:27:55.533 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:27:55.534 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:27:55.534 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:27:55.534 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:27:55.534 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:27:55.534 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:55.535 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:55.535 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:27:55.535 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:27:55.535 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:27:55.535 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:27:55.536 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:27:55.536 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:27:55.536 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:27:55.537 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.488676_136.1 2023-10-12T19:28:00.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:00.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:00.540 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:00.634 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:00.634 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:01.043 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:01.044 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:01.044 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:01.044 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:01.045 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:01.045 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:01.045 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:01.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:01.046 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:01.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:01.046 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:01.046 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:01.046 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:01.047 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:28:01.047 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:01.047 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:01.047 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:01.047 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:28:01.048 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:01.048 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:01.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:01.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:01.048 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:01.049 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:01.049 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:01.049 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:01.049 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:01.049 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:01.050 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:01.050 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:01.050 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:01.050 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:01.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:01.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:01.051 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:01.051 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:01.051 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:01.051 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:01.052 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:01.052 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:01.052 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:01.052 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:01.052 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:01.053 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:01.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:01.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:01.053 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:01.053 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:01.054 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:01.054 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:01.054 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:01.054 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:01.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:01.054 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:01.055 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:01.055 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:01.055 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:01.055 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:01.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:01.056 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:01.056 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:01.056 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:01.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:01.056 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:01.057 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:28:06.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:06.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:06.060 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:06.104 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:06.104 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:06.158 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:06.158 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:06.158 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:06.158 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:06.159 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:06.159 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:06.159 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:06.159 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:06.159 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:06.160 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:06.160 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:06.160 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:06.160 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:06.160 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:28:06.161 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:06.161 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:06.161 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:06.161 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:28:06.161 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:06.162 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:06.162 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:06.162 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:06.163 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:06.163 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:06.164 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:06.164 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:06.164 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:06.165 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:06.165 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:06.165 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:06.166 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:06.166 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:06.167 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:06.167 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:06.167 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:06.168 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:06.168 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:06.169 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:06.169 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:06.170 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:06.170 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:06.170 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:06.171 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:06.171 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:06.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:06.172 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:06.172 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:06.172 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:06.173 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:06.173 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:06.173 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:06.174 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:06.174 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:06.174 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:06.175 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:06.175 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:06.175 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:06.176 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:06.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:06.176 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:06.177 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:06.177 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:06.178 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:06.178 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:06.180 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:28:11.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:11.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:11.183 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:11.236 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:11.236 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:11.299 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:28:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:28:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:11.312 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:11.313 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:28:16.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:16.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:16.315 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:16.358 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:16.359 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:16.411 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:16.411 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:16.411 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:16.411 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:16.412 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:16.412 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:16.412 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:16.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:16.413 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:16.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:16.413 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:16.413 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:16.413 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:16.414 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:28:16.414 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:16.414 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:16.414 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:16.414 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:28:16.415 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:16.415 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:16.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:16.415 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:16.415 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:16.416 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:16.416 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:16.416 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:16.416 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:16.416 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:16.417 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:16.417 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:16.417 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:16.417 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:16.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:16.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:16.418 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:16.418 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:16.418 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:16.418 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:16.418 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:16.419 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:16.419 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:16.419 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:16.419 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:16.419 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:16.420 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:16.420 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:16.420 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:16.420 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:16.420 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:16.420 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:16.421 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:16.421 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:16.421 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:16.421 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:16.421 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:16.422 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:16.422 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:16.422 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:16.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:16.423 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:16.423 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:16.423 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:16.424 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:16.424 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:16.426 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:28:21.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:21.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:21.429 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:21.479 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:21.479 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:21.545 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:21.546 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:21.546 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:21.546 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:21.546 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:21.546 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:21.547 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:21.547 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:21.547 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:21.547 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:21.547 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:21.548 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:21.548 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:21.548 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.571704_145.1", 2023-10-12T19:28:21.548 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:21.548 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:21.548 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:21.549 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.571704Z", 2023-10-12T19:28:21.549 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:21.549 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:21.549 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:21.549 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:21.551 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:21.551 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:21.551 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:21.551 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:21.551 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:21.552 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:21.552 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:21.552 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:21.552 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:21.552 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:21.552 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:21.553 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:21.553 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:21.553 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:21.553 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:21.554 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:21.554 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:21.554 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:21.555 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:21.555 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:21.555 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:21.555 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:21.555 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:21.556 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:21.556 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:21.557 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:21.557 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:21.557 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:21.557 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:21.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:21.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:21.558 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:21.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:21.559 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:21.559 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:21.559 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:21.559 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:21.559 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:21.560 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.571704_145.1 2023-10-12T19:28:26.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:26.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:26.563 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:26.648 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:26.648 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:26.837 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:26.838 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:26.838 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:26.839 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:26.839 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:26.839 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:26.839 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:26.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:26.840 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:26.840 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:26.840 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:26.841 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:26.841 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:26.841 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:28:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:28:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:26.845 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:26.845 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:26.845 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:26.845 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:26.848 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:26.848 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:26.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:26.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:26.849 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:26.849 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:26.849 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:26.849 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:26.850 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:26.850 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:26.850 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:26.850 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:26.850 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:26.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:26.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:26.851 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:26.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:26.852 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:26.852 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:26.852 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:26.852 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:26.852 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:26.854 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:28:31.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:31.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:31.856 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:31.929 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:31.929 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:32.383 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:32.383 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:32.383 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:32.383 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:32.384 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:32.384 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:32.384 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:32.384 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:32.384 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:32.385 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:32.385 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:32.385 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:32.385 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:32.385 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470052_132.1", 2023-10-12T19:28:32.386 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:32.386 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:32.386 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:32.386 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470052Z", 2023-10-12T19:28:32.387 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:32.387 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:32.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:32.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:32.387 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:32.388 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:32.388 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:32.388 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:32.388 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:32.388 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:32.389 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:32.389 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:32.389 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:32.389 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:32.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:32.389 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:32.390 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:32.390 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:32.390 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:32.390 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:32.390 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:32.391 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:32.391 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:32.391 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:32.391 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:32.391 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:32.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:32.392 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:32.392 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:32.392 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:32.392 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:32.393 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:32.393 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:32.393 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:32.393 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:32.393 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:32.393 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:32.394 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:32.394 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:32.394 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:32.394 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:32.395 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:32.395 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:32.395 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:32.395 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:32.395 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:32.396 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.470052_132.1 2023-10-12T19:28:37.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:37.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:37.399 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:37.445 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:37.445 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:37.533 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:37.533 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:37.533 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:37.534 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:37.534 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:37.534 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:37.534 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:37.535 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:37.535 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:37.535 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:37.535 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:37.535 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:37.536 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:37.536 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:28:37.536 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:37.536 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:37.536 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:37.537 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:28:37.537 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:37.537 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:37.537 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:37.537 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:37.538 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:37.538 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:37.538 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:37.538 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:37.538 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:37.539 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:37.539 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:37.539 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:37.539 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:37.539 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:37.540 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:37.540 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:37.540 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:37.540 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:37.540 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:37.541 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:37.541 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:37.541 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:37.541 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:37.541 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:37.542 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:37.542 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:37.542 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:37.542 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:37.542 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:37.543 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:37.543 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:37.543 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:37.543 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:37.543 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:37.544 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:37.544 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:37.544 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:37.544 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:37.544 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:37.545 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:37.545 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:37.545 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:37.545 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:37.545 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:37.545 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:37.546 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:37.547 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.470944_133.1 2023-10-12T19:28:42.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:42.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:42.549 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:42.631 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:42.632 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:42.959 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:42.959 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:42.960 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:42.960 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:42.960 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:42.960 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:42.960 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:42.961 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:42.961 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:42.961 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:42.961 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:42.961 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:42.961 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:42.962 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:28:42.962 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:42.962 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:42.962 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:28:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:42.963 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:42.964 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:42.964 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:42.964 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:42.964 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:42.964 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:42.964 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:42.966 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:42.966 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:42.966 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:42.966 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:42.966 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:42.967 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:42.970 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:42.970 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:42.970 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:42.970 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:42.970 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:42.971 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:42.971 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:42.971 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:42.971 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:42.971 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:42.972 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.436071_129.1 2023-10-12T19:28:47.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:47.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:47.975 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:48.076 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:48.076 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:48.377 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:48.378 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:48.378 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:48.378 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:48.378 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:48.379 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:48.379 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:48.379 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:48.379 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:48.379 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:48.380 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:48.380 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:48.380 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:48.380 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:28:48.380 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:48.381 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:48.381 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:48.381 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:28:48.381 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:48.382 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:48.382 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:48.382 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:48.382 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:48.382 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:48.383 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:48.383 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:48.383 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:48.383 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:48.383 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:48.384 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:48.384 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:48.384 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:48.384 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:48.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:48.385 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:48.385 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:48.385 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:48.385 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:48.385 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:48.385 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:48.386 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:48.386 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:48.386 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:48.386 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:48.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:48.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:48.387 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:48.387 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:48.387 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:48.388 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:48.388 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:48.388 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:48.388 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:48.388 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:48.388 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:48.389 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:48.389 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:48.389 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:48.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:48.389 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:48.390 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:48.390 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:48.390 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:48.390 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:48.391 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.470944_133.1 2023-10-12T19:28:53.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:53.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:53.393 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:53.441 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:53.441 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:53.496 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:53.497 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:53.497 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:53.497 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:53.497 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:53.498 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:53.498 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:53.498 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:53.498 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:53.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:53.499 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:53.499 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:53.499 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:53.500 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:28:53.500 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:53.500 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:53.500 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:53.500 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:28:53.501 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:53.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:53.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:53.501 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:53.501 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:53.502 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:53.502 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:53.502 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:53.502 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:53.502 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:53.503 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:53.503 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:53.503 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:53.503 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:53.503 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:53.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:53.504 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:53.504 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:53.504 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:53.504 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:53.504 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:53.505 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:53.505 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:53.505 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:53.505 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:53.505 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:53.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:53.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:53.506 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:53.506 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:53.506 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:53.507 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:53.507 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:53.507 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:53.507 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:53.508 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:53.508 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:53.508 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:53.508 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:53.508 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:53.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:53.509 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:53.509 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:53.509 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:53.509 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:53.509 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:53.511 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.470944_133.1 2023-10-12T19:28:58.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:58.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:28:58.514 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:28:58.557 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:28:58.557 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:28:58.611 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:28:58.611 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:28:58.611 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:28:58.611 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:28:58.612 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:28:58.612 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:28:58.612 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:58.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:58.612 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:28:58.613 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:58.613 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:28:58.613 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:58.613 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:58.613 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:28:58.614 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:58.614 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:28:58.614 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:58.614 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:28:58.614 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:58.615 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:58.615 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:58.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:58.615 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:28:58.615 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:58.616 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:58.616 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:28:58.616 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:58.616 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:58.616 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:58.617 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:28:58.617 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:58.617 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:58.617 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:58.617 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:58.617 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:28:58.618 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:58.618 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:58.618 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:58.618 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:58.618 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:58.619 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:58.619 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:58.619 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:58.619 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:58.620 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:58.620 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:28:58.620 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:28:58.620 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:28:58.620 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:28:58.620 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:28:58.621 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:28:58.621 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:28:58.621 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:28:58.621 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:28:58.621 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:28:58.622 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:58.622 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:58.622 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:28:58.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:28:58.622 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:28:58.623 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:28:58.623 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:28:58.623 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:28:58.623 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:28:58.624 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.407338_124.1 2023-10-12T19:29:03.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:03.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:03.627 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:03.670 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:03.670 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:03.724 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:03.724 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:03.725 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:03.725 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:03.725 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:03.725 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:03.725 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:03.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:03.726 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:03.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:03.726 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:03.726 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:03.727 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:03.727 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:29:03.727 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:03.727 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:03.727 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:03.728 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:29:03.728 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:03.728 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:03.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:03.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:03.728 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:03.729 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:03.729 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:03.729 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:03.729 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:03.729 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:03.730 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:03.730 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:03.730 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:03.730 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:03.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:03.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:03.731 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:03.731 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:03.731 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:03.731 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:03.732 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:03.732 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:03.732 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:03.732 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:03.732 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:03.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:03.734 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:03.734 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:03.734 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:03.734 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:03.734 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:03.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:03.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:03.735 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:03.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:03.735 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:03.736 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:03.736 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:03.736 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:03.736 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:03.737 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.470944_133.1 2023-10-12T19:29:08.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:08.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:08.740 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:08.950 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:08.950 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:08.951 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:08.951 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:08.951 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:08.951 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:08.952 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:08.952 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:08.952 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:08.952 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:08.952 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:08.952 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:08.953 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:08.953 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:29:08.953 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:08.953 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:08.953 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:08.954 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:29:08.954 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:08.954 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:08.954 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:08.954 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:08.955 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:08.955 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:08.955 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:08.955 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:08.955 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:08.956 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:08.956 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:08.956 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:08.956 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:08.956 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:08.957 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:08.957 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:08.957 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:08.957 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:08.957 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:08.958 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:08.958 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:08.958 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:08.958 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:08.959 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:08.959 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:08.959 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:08.959 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:08.960 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:08.960 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:08.960 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:08.960 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:08.960 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:08.961 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:08.961 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:08.961 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:08.961 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:08.962 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:08.962 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:08.962 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:08.963 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:08.963 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:08.963 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:08.963 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:08.963 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:08.964 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:08.964 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:08.965 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.407338_124.1 2023-10-12T19:29:13.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:13.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:13.967 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:14.064 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:14.065 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:14.120 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:14.120 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:14.120 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:14.120 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:14.121 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:14.121 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:14.121 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:14.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:14.122 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:14.122 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:14.122 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:14.122 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:14.122 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:14.123 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:29:14.123 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:14.123 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:14.123 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:14.123 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:29:14.124 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:14.124 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:14.124 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:14.124 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:14.124 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:14.125 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:14.125 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:14.125 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:14.125 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:14.125 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:14.125 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:14.126 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:14.126 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:14.126 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:14.126 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:14.126 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:14.127 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:14.127 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:14.127 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:14.127 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:14.127 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:14.128 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:14.128 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:14.128 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:14.128 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:14.128 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:14.130 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:14.130 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:14.130 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:14.130 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:14.130 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:14.131 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:14.131 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:14.131 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:14.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:14.131 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:14.132 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:14.132 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:14.132 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:14.132 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:14.133 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:29:19.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:19.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:19.135 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:19.179 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:19.179 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:19.236 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:19.236 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:19.236 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:19.236 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:19.237 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:19.237 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:19.237 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:19.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:19.237 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:19.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:19.238 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:19.238 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:19.238 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:19.238 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:29:19.239 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:19.239 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:19.239 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:19.239 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:29:19.239 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:19.240 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:19.240 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:19.240 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:19.240 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:19.241 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:19.241 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:19.241 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:19.242 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:19.242 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:19.242 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:19.242 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:19.242 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:19.243 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:19.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:19.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:19.243 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:19.243 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:19.244 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:19.244 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:19.244 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:19.244 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:19.244 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:19.244 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:19.245 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:19.245 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:19.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:19.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:19.245 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:19.246 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:19.246 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:19.246 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:19.246 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:19.246 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:19.247 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:19.247 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:19.247 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:19.247 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:19.247 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:19.248 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:19.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:19.248 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:19.248 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:19.248 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:19.248 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:19.249 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:19.250 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:29:24.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:24.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:24.252 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:24.295 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:24.296 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:24.349 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:24.350 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:24.350 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:24.350 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:24.350 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:24.350 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:24.351 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:24.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:24.351 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:24.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:24.351 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:24.352 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:24.352 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:24.352 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.571704_145.1", 2023-10-12T19:29:24.352 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:24.352 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:24.354 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:24.354 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.571704Z", 2023-10-12T19:29:24.355 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:24.355 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:24.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:24.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:24.355 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:24.356 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:24.356 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:24.356 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:24.356 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:24.356 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:24.357 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:24.357 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:24.357 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:24.357 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:24.357 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:24.358 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:24.358 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:24.358 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:24.358 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:24.358 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:24.359 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:24.359 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:24.359 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:24.359 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:24.359 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:24.359 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:24.360 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:24.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:24.360 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:24.360 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:24.360 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:24.361 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:24.361 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:24.361 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:24.361 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:24.361 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:24.362 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:24.362 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:24.362 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:24.362 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:24.362 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:24.363 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:24.363 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:24.363 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:24.363 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:24.363 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:24.364 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.571704_145.1 2023-10-12T19:29:29.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:29.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:29.367 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:29.410 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:29.411 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:29.489 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:29.490 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:29.490 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:29.490 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:29.490 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:29.490 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:29.491 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:29.491 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:29.491 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:29.491 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:29.491 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:29.492 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:29.492 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:29.492 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.769932_175.1", 2023-10-12T19:29:29.492 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:29.492 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:29.493 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:29.493 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.769932Z", 2023-10-12T19:29:29.493 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:29.493 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:29.493 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:29.493 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:29.494 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:29.494 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:29.494 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:29.494 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:29.494 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:29.495 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:29.495 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:29.495 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:29.495 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:29.495 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:29.496 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:29.496 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:29.496 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:29.496 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:29.496 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:29.497 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:29.497 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:29.497 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:29.497 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:29.497 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:29.498 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:29.498 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:29.498 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:29.498 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:29.498 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:29.499 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:29.499 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:29.499 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:29.499 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:29.499 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:29.499 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:29.500 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:29.500 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:29.500 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:29.500 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:29.500 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:29.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:29.501 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:29.501 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:29.501 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:29.501 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:29.501 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:29.502 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.769932_175.1 2023-10-12T19:29:34.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:34.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:34.505 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:34.578 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:34.578 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:34.982 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:34.983 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:34.983 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:34.984 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:34.984 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:34.984 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:34.984 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:34.985 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:34.985 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:34.985 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:34.986 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:34.986 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:34.986 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:34.987 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:29:34.987 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:34.987 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:34.988 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:34.988 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:29:34.988 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:34.989 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:34.989 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:34.989 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:34.990 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:34.990 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:34.990 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:34.991 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:34.991 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:34.991 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:34.991 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:34.992 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:34.992 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:34.992 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:34.993 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:34.993 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:34.993 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:34.994 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:34.994 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:34.994 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:34.995 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:34.995 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:34.995 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:34.996 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:34.996 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:34.996 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:34.997 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:34.997 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:34.997 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:34.998 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:34.998 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:34.998 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:34.999 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:34.999 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:34.999 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:35.000 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:35.000 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:35.000 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:35.001 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:35.001 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:35.001 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:35.002 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:35.002 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:35.002 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:35.003 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:35.003 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:35.005 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:29:40.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:40.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:40.007 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:40.050 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:40.051 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:40.113 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:40.113 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:40.113 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:40.113 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:40.114 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:40.114 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:40.114 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:40.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:40.114 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:40.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:40.115 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:40.115 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:40.115 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:40.115 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:29:40.116 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:40.116 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:40.116 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:40.116 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:29:40.116 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:40.117 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:40.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:40.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:40.117 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:40.117 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:40.118 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:40.118 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:40.118 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:40.118 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:40.118 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:40.119 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:40.119 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:40.119 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:40.119 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:40.119 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:40.120 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:40.120 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:40.120 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:40.120 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:40.120 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:40.121 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:40.121 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:40.121 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:40.121 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:40.121 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:40.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:40.122 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:40.122 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:40.126 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:40.127 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:40.127 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:40.129 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:40.129 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:40.130 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:40.130 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:40.130 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:40.130 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:40.131 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:40.131 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:40.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:40.133 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:40.134 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:40.134 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:40.134 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:40.135 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:40.136 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.467824_131.1 2023-10-12T19:29:45.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:45.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:45.140 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:45.232 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:45.232 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:45.514 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:45.515 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:45.515 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:45.515 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:45.515 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:45.516 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:45.516 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:45.516 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:45.516 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:45.516 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:45.517 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:45.517 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:45.517 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:45.517 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:29:45.517 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:45.518 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:45.518 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:45.518 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:29:45.518 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:45.518 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:45.519 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:45.519 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:45.519 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:45.519 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:45.519 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:45.520 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:45.520 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:45.520 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:45.520 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:45.521 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:45.521 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:45.521 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:45.521 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:45.521 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:45.521 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:45.522 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:45.522 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:45.522 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:45.522 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:45.522 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:45.523 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:45.523 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:45.523 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:45.523 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:45.523 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:45.524 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:45.524 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:45.524 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:45.524 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:45.524 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:45.525 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:45.525 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:45.525 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:45.525 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:45.525 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:45.525 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:45.526 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:45.526 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:45.526 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:45.526 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:45.526 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:45.527 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:45.527 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:45.527 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:45.528 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:29:50.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:50.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:50.532 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:50.645 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:50.645 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:50.924 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:50.925 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:50.925 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:50.925 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:50.925 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:50.926 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:50.926 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:50.926 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:50.926 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:50.926 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:50.927 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:50.927 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:50.927 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:50.927 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:29:50.927 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:50.928 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:50.928 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:50.928 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:29:50.928 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:50.928 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:50.928 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:50.929 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:50.929 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:50.929 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:50.929 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:50.929 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:50.930 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:50.930 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:50.930 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:50.930 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:50.930 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:50.931 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:50.931 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:50.931 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:50.931 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:50.931 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:50.931 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:50.932 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:50.932 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:50.932 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:50.932 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:50.932 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:50.933 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:50.933 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:50.933 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:50.933 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:50.933 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:50.934 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:50.934 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:50.934 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:50.934 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:50.934 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:50.935 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:50.935 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:50.935 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:50.935 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:50.935 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:50.935 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:50.936 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:50.936 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:50.936 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:50.936 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:50.936 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:50.937 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:50.938 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:29:55.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:55.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:29:55.940 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:29:56.040 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:29:56.040 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:29:56.470 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:29:56.471 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:29:56.471 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:29:56.471 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:29:56.471 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:29:56.472 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:29:56.472 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:56.472 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:56.472 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:29:56.472 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:56.473 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:29:56.473 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:56.473 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:56.473 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:29:56.473 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:56.474 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:29:56.474 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:56.474 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:29:56.474 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:56.474 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:56.474 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:56.475 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:56.475 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:29:56.475 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:56.475 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:56.475 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:29:56.476 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:56.476 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:56.476 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:56.476 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:29:56.476 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:56.477 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:56.477 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:56.477 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:56.477 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:29:56.477 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:56.477 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:56.478 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:56.478 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:56.478 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:56.478 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:56.478 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:56.479 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:56.479 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:56.479 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:56.479 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:29:56.479 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:29:56.480 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:29:56.480 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:29:56.480 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:29:56.480 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:29:56.480 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:29:56.481 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:29:56.481 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:29:56.481 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:29:56.481 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:56.481 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:56.481 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:29:56.482 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:29:56.482 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:29:56.482 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:29:56.482 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:29:56.482 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:29:56.483 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:29:56.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697138686.129941_308.1 target=1_1697136751.408247_125.1 2023-10-12T19:30:01.486 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2023-10-12T19:30:01.487 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-50 2023-10-12T19:30:01.509 INFO:rgw_multi.tests:create bucket zone=a2 name=pqrzfz-51 2023-10-12T19:30:03.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T19:30:03.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pqrzfz-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T19:30:03.217 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pqrzfz-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T19:30:03.266 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:03.266 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:03.306 INFO:teuthology.orchestra.run.smithi170.stderr:could not get bucket info for bucket=pqrzfz-50 2023-10-12T19:30:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-12T19:30:03.321 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... FAIL 2023-10-12T19:30:03.322 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-52 2023-10-12T19:30:03.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:03.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:03.420 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:03.463 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:03.463 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:03.502 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T19:30:03.517 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T19:30:03.518 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:03.518 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697139003.336876_324.1", 2023-10-12T19:30:03.518 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T19:30:03.336876Z" 2023-10-12T19:30:03.518 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:03.518 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:03.519 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:03.519 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T19:30:03.519 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:03.519 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:03.519 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:03.520 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:30:03.520 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:03.520 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:03.520 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:03.520 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:30:03.521 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:03.521 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T19:30:03.522 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T19:30:03.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:03.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:03.524 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:03.606 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:03.606 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:03.659 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:03.659 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:03.660 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:03.660 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:03.660 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:03.660 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:03.661 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:03.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:03.661 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:03.661 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:03.662 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:03.662 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:03.662 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:03.662 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.486122_135.1", 2023-10-12T19:30:03.662 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:03.663 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:03.663 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:03.663 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.486122Z", 2023-10-12T19:30:03.663 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:03.663 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:03.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:03.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:03.664 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:03.664 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:03.664 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:03.665 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:03.665 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:03.665 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:03.665 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:03.665 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:03.666 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:03.666 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:03.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:03.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:03.667 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:03.667 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:03.667 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:03.667 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:03.667 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:03.668 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:03.668 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:03.668 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:03.668 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:03.668 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:03.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:03.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:03.669 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:03.669 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:03.670 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:03.670 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:03.670 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:03.670 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:03.671 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:03.671 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:03.671 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:03.671 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:03.671 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:03.672 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:03.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:03.672 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:03.672 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:03.672 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:03.673 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:03.673 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:03.674 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.486122_135.1 2023-10-12T19:30:08.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:08.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:08.677 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:08.724 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:08.724 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:08.779 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:30:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:30:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:08.786 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:08.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:08.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:08.786 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:08.786 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:08.787 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:08.787 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:08.787 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:08.787 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:08.787 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:08.787 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:08.788 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:08.788 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:08.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:08.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:08.788 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:08.789 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:08.789 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:08.789 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:08.789 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:08.790 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:08.790 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:08.790 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:08.790 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:08.791 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:08.791 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:08.791 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:08.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:08.791 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:08.791 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:08.792 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:08.792 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:08.792 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:08.793 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.470944_133.1 2023-10-12T19:30:13.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:13.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:13.796 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:13.838 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:13.838 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:13.925 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:13.926 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:13.926 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:13.926 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:13.927 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:13.927 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:13.927 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:13.927 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:13.927 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:13.928 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:13.928 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:13.928 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:13.928 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:13.928 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:30:13.929 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:13.929 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:13.929 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:13.929 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:30:13.929 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:13.930 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:13.930 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:13.930 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:13.930 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:13.930 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:13.931 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:13.931 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:13.931 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:13.931 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:13.932 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:13.932 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:13.932 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:13.932 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:13.933 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:13.933 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:13.933 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:13.933 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:13.933 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:13.934 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:13.934 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:13.934 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:13.934 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:13.934 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:13.935 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:13.935 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:13.935 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:13.935 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:13.935 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:13.936 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:13.936 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:13.936 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:13.936 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:13.936 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:13.937 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:13.937 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:13.937 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:13.937 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:13.937 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:13.938 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:13.938 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:13.938 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:13.938 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:13.939 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:13.939 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:13.939 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:13.941 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:30:18.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:18.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:18.944 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:19.009 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:19.009 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:19.390 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:19.390 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:19.390 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:19.391 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:19.391 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:19.391 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:19.392 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:19.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:19.392 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:19.392 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:19.392 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:19.392 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:19.393 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:19.393 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:30:19.393 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:19.393 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:19.393 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:19.394 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:30:19.394 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:19.394 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:19.394 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:19.394 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:19.395 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:19.395 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:19.395 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:19.395 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:19.395 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:19.396 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:19.396 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:19.396 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:19.396 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:19.396 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:19.396 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:19.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:19.397 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:19.397 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:19.397 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:19.397 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:19.398 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:19.398 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:19.398 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:19.398 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:19.398 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:19.399 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:19.399 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:19.399 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:19.399 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:19.399 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:19.400 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:19.400 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:19.400 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:19.400 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:19.400 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:19.401 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:19.401 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:19.401 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:19.402 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:19.402 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:19.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:19.402 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:19.403 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:19.403 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:19.403 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:19.403 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:19.404 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.407338_124.1 2023-10-12T19:30:24.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:24.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:24.407 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:24.452 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:24.452 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:24.507 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:24.508 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:24.508 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:24.508 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:24.509 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:24.509 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:24.509 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:24.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:24.510 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:24.510 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:24.510 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:24.510 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:24.511 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:24.511 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:30:24.511 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:24.511 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:24.511 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:24.512 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:30:24.512 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:24.512 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:24.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:24.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:24.513 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:24.513 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:24.513 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:24.513 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:24.513 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:24.514 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:24.514 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:24.514 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:24.514 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:24.514 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:24.514 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:24.515 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:24.515 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:24.515 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:24.515 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:24.515 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:24.516 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:24.516 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:24.516 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:24.516 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:24.517 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:24.517 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:24.517 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:24.517 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:24.518 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:24.518 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:24.518 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:24.518 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:24.519 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:24.519 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:24.519 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:24.519 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:24.519 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:24.520 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:24.520 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:24.520 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:24.520 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:24.520 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:24.521 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:24.521 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:24.521 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:24.521 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:24.523 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:30:29.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:29.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:29.525 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:29.618 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:29.618 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:29.852 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:29.852 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:29.852 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:29.852 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:29.853 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:29.853 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:29.853 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:29.853 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:29.853 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:29.854 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:29.854 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:29.854 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:29.854 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:29.854 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:30:29.855 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:29.855 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:29.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:29.855 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:30:29.855 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:29.856 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:29.856 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:29.856 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:29.856 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:29.856 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:29.857 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:29.857 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:29.857 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:29.857 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:29.858 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:29.858 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:29.858 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:29.858 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:29.858 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:29.859 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:29.859 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:29.859 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:29.859 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:29.859 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:29.859 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:29.860 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:29.860 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:29.860 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:29.860 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:29.860 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:29.861 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:29.861 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:29.861 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:29.861 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:29.861 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:29.861 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:29.862 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:29.862 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:29.862 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:29.862 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:29.863 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:29.863 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:29.863 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:29.863 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:29.863 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:29.863 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:29.864 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:29.864 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:29.864 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:29.864 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:29.865 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:30:34.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:34.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:34.868 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:34.949 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:34.950 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:35.319 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:35.320 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:35.320 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:35.320 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:35.320 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:35.321 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:35.321 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:35.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:35.321 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:35.321 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:35.322 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:35.322 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:35.322 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:35.322 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:30:35.322 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:35.323 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:35.323 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:35.323 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:30:35.323 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:35.323 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:35.324 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:35.324 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:35.324 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:35.324 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:35.324 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:35.325 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:35.325 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:35.325 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:35.325 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:35.325 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:35.326 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:35.326 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:35.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:35.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:35.326 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:35.327 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:35.327 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:35.327 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:35.327 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:35.327 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:35.328 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:35.328 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:35.328 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:35.328 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:35.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:35.329 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:35.329 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:35.329 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:35.329 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:35.329 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:35.330 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:35.330 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:35.330 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:35.330 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:35.330 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:35.330 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:35.331 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:35.331 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:35.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:35.331 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:35.331 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:35.332 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:35.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:35.332 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:35.332 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.407338_124.1 2023-10-12T19:30:40.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:40.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:40.335 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:40.378 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:40.378 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:40.433 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:40.433 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:40.434 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:40.434 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:40.435 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:40.435 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:40.435 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:40.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:30:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:30:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:40.439 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:40.439 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:40.439 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:40.439 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:40.440 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:40.440 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:40.440 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:40.442 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:40.442 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:40.443 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:40.443 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:40.443 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:40.444 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:40.444 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:40.444 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:40.444 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:40.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:40.448 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:40.448 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:40.448 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:40.448 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:40.449 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.435183_128.1 2023-10-12T19:30:45.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:45.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:45.452 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:45.494 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:45.495 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:45.546 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:45.546 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:45.547 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:45.547 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:45.547 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:45.547 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:45.547 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:45.548 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:45.548 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:45.548 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:45.548 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:45.548 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:45.549 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:45.549 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:30:45.549 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:45.549 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:45.549 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:45.550 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:30:45.550 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:45.550 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:45.550 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:45.550 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:45.551 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:45.551 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:45.551 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:45.551 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:45.551 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:45.552 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:45.552 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:45.552 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:45.552 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:45.552 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:45.553 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:45.553 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:45.553 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:45.553 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:45.553 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:45.553 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:45.554 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:45.554 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:45.554 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:45.554 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:45.554 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:45.555 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:45.555 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:45.555 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:45.555 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:45.555 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:45.556 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:45.556 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:45.556 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:45.556 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:45.556 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:45.557 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:45.557 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:45.557 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:45.557 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:45.557 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:45.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:45.558 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:45.558 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:45.558 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:45.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:45.559 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:45.560 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:30:50.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:50.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:50.562 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:50.606 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:50.606 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:50.664 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:50.664 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:50.665 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:50.665 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:50.665 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:50.665 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:50.665 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:50.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:50.666 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:50.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:50.666 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:50.666 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:50.667 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:50.667 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:30:50.667 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:50.667 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:50.667 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:50.668 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:30:50.668 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:50.668 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:50.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:50.668 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:50.669 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:50.669 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:50.669 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:50.669 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:50.669 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:50.669 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:50.670 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:50.670 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:50.670 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:50.670 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:50.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:50.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:50.671 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:50.671 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:50.671 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:50.672 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:50.672 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:50.672 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:50.672 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:50.672 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:50.673 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:50.673 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:50.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:50.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:50.673 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:50.673 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:50.674 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:50.674 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:50.674 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:50.674 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:50.674 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:50.675 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:50.675 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:50.675 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:50.675 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:50.675 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:50.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:50.676 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:50.676 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:50.676 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:50.676 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:50.676 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:50.678 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:30:55.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:55.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:30:55.682 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:30:55.726 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:30:55.726 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:30:55.784 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:30:55.784 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:30:55.785 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:30:55.785 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:30:55.785 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:30:55.785 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:30:55.785 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:55.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:55.786 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:30:55.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:55.786 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:30:55.787 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:55.787 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:55.787 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:30:55.787 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:55.787 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:30:55.788 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:55.788 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:30:55.788 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:55.788 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:55.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:55.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:55.789 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:30:55.789 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:55.789 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:55.789 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:30:55.789 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:55.790 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:55.790 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:55.790 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:30:55.790 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:55.790 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:55.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:55.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:55.791 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:30:55.791 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:55.791 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:55.792 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:55.792 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:55.792 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:55.792 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:55.792 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:55.793 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:55.793 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:55.793 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:55.793 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:30:55.793 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:30:55.794 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:30:55.794 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:30:55.794 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:30:55.794 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:30:55.794 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:30:55.795 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:30:55.795 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:30:55.795 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:30:55.795 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:55.795 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:55.795 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:30:55.796 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:30:55.796 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:30:55.796 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:30:55.796 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:30:55.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:30:55.797 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:30:55.799 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:31:00.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:00.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:00.804 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:00.882 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:00.882 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:01.285 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:01.285 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:01.285 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:01.286 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:01.286 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:01.286 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:01.286 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:01.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:01.287 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:01.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:01.287 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:01.287 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:01.288 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:01.288 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:31:01.288 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:01.288 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:01.288 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:01.289 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:31:01.289 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:01.289 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:01.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:01.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:01.290 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:01.290 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:01.290 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:01.290 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:01.290 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:01.291 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:01.291 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:01.291 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:01.291 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:01.291 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:01.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:01.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:01.292 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:01.292 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:01.292 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:01.293 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:01.293 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:01.293 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:01.293 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:01.293 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:01.294 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:01.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:01.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:01.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:01.294 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:01.295 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:01.295 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:01.295 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:01.295 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:01.295 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:01.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:01.297 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:01.298 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:01.299 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:31:06.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:06.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:06.303 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:06.382 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:06.383 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:06.457 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:06.457 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:06.458 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:06.458 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:06.458 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:06.458 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:06.459 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:06.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:06.459 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:06.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:06.459 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:06.460 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:06.460 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:06.460 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:31:06.460 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:06.460 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:06.461 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:06.461 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:31:06.461 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:06.461 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:06.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:06.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:06.462 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:06.462 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:06.462 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:06.462 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:06.463 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:06.463 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:06.463 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:06.464 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:06.464 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:06.464 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:06.464 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:06.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:06.465 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:06.465 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:06.465 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:06.465 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:06.466 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:06.466 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:06.466 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:06.466 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:06.467 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:06.467 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:06.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:06.467 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:06.467 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:06.468 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:06.468 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:06.468 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:06.468 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:06.468 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:06.469 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:06.469 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:06.469 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:06.469 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:06.469 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:06.470 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:06.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:06.470 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:06.470 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:06.470 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:06.471 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:06.471 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:06.472 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:31:11.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:11.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:11.474 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:11.555 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:11.556 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:11.881 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:11.881 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:11.881 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:11.882 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:11.882 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:11.882 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:11.882 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:11.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:11.883 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:11.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:11.883 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:11.883 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:11.883 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:11.884 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:31:11.884 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:11.884 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:11.884 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:11.884 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:31:11.885 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:11.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:11.885 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:11.885 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:11.885 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:11.886 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:11.886 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:11.886 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:11.886 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:11.886 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:11.887 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:11.887 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:11.887 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:11.887 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:11.887 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:11.888 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:11.888 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:11.888 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:11.888 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:11.888 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:11.889 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:11.889 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:11.889 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:11.889 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:11.889 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:11.890 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:11.890 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:11.890 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:11.890 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:11.890 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:11.891 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:11.891 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:11.891 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:11.891 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:11.891 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:11.892 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:11.892 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:11.892 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:11.892 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:11.892 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:11.892 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:11.893 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:11.893 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:11.893 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:11.893 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:11.893 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:11.894 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.407338_124.1 2023-10-12T19:31:16.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:16.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:16.897 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:16.939 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:16.940 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:16.993 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:16.993 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:16.993 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:16.994 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:16.994 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:16.994 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:16.994 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:16.995 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:16.995 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:16.995 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:16.995 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:16.995 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:16.996 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:16.996 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:31:16.996 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:16.996 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:16.997 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:16.997 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:31:16.997 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:16.997 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:16.997 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:16.998 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:16.998 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:16.998 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:16.998 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:16.999 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:16.999 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:16.999 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:16.999 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:17.000 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:17.000 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:17.000 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:17.000 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:17.001 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:17.001 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:17.001 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:17.001 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:17.001 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:17.002 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:17.002 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:17.002 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:17.002 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:17.003 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:17.003 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:17.003 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:17.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:17.004 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:17.004 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:17.004 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:17.004 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:17.004 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:17.005 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:17.005 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:17.005 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:17.005 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:17.005 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:17.006 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:17.006 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:17.006 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:17.006 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:17.006 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:17.007 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:17.007 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:17.007 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:17.008 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.432502_127.1 2023-10-12T19:31:22.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:22.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:22.011 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:22.054 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:22.054 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:22.105 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:22.105 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:22.105 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:22.106 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:22.106 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:22.106 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:22.106 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:22.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:22.107 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:22.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:22.107 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:22.107 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:22.107 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:22.108 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:31:22.108 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:22.108 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:22.108 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:22.108 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:31:22.109 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:22.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:22.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:22.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:22.109 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:22.110 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:22.110 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:22.110 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:22.110 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:22.110 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:22.111 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:22.111 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:22.111 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:22.111 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:22.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:22.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:22.112 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:22.112 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:22.112 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:22.112 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:22.113 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:22.113 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:22.113 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:22.114 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:22.114 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:22.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:22.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:22.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:22.115 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:22.115 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:22.115 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:22.115 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:22.115 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:22.116 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:22.116 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:22.116 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:22.116 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:22.116 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:22.117 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:22.117 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:22.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:22.117 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:22.117 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:22.117 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:22.118 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:22.118 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:22.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.432502_127.1 2023-10-12T19:31:27.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:27.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:27.122 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:27.164 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:27.164 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:27.214 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:27.215 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:27.215 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:27.215 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:27.216 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:27.216 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:27.216 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:27.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:27.216 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:27.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:27.217 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:27.217 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:27.217 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:27.217 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:31:27.217 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:27.218 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:27.218 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:27.218 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:31:27.218 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:27.218 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:27.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:27.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:27.219 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:27.219 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:27.219 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:27.220 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:27.220 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:27.220 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:27.220 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:27.220 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:27.221 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:27.221 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:27.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:27.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:27.221 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:27.222 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:27.222 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:27.222 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:27.222 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:27.222 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:27.223 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:27.223 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:27.223 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:27.223 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:27.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:27.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:27.224 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:27.224 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:27.224 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:27.224 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:27.225 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:27.225 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:27.225 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:27.225 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:27.225 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:27.226 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:27.226 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:27.226 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:27.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:27.226 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:27.227 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:27.227 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:27.227 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:27.227 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:27.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.407338_124.1 2023-10-12T19:31:32.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:32.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:32.231 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:32.274 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:32.275 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:32.330 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:32.331 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:32.331 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:32.331 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:32.331 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:32.332 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:32.332 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:32.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:32.332 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:32.333 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:31:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:31:32.334 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:32.335 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:32.336 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:32.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:32.338 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:32.339 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:32.340 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:32.341 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:32.342 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:32.344 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:32.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.436071_129.1 2023-10-12T19:31:37.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:37.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:37.348 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:37.422 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:37.423 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:37.510 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:37.511 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:37.511 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:37.512 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:37.512 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:37.512 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:37.512 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:37.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:37.513 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:37.513 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:37.513 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:37.513 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:37.514 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:37.514 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:31:37.514 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:37.514 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:37.514 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:37.515 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:31:37.515 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:37.515 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:37.515 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:37.515 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:37.516 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:37.516 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:37.516 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:37.516 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:37.516 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:37.517 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:37.517 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:37.517 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:37.517 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:37.517 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:37.517 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:37.518 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:37.518 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:37.518 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:37.518 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:37.518 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:37.519 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:37.519 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:37.519 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:37.519 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:37.519 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:37.520 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:37.520 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:37.520 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:37.520 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:37.521 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:37.521 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:37.521 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:37.521 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:37.522 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:37.522 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:37.522 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:37.522 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:37.522 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:37.523 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:37.523 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:37.523 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:37.523 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:37.523 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:37.524 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:37.524 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:37.524 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:37.525 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.407338_124.1 2023-10-12T19:31:42.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:42.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:42.528 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:42.575 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:42.575 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:42.628 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:42.628 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:42.628 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:42.629 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:42.629 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:42.629 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:42.629 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:42.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:42.630 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:42.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:42.630 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:42.630 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:42.631 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:42.631 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.680443_163.1", 2023-10-12T19:31:42.631 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:42.631 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:42.632 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:42.632 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.680443Z", 2023-10-12T19:31:42.632 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:42.632 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:42.632 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:42.633 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:42.633 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:42.633 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:42.633 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:42.634 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:42.634 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:42.634 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:42.634 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:42.635 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:42.635 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:42.635 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:42.635 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:42.635 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:42.635 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:42.636 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:42.636 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:42.636 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:42.636 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:42.636 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:42.637 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:42.637 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:42.637 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:42.637 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:42.637 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:42.638 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:42.638 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:42.638 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:42.638 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:42.638 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:42.639 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:42.639 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:42.639 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:42.639 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:42.639 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:42.639 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:42.640 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:42.640 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:42.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:42.640 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:42.641 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:42.641 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:42.641 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:42.641 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:42.642 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.680443_163.1 2023-10-12T19:31:47.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:47.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:47.645 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:47.724 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:47.724 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:48.035 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:48.035 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:48.035 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:48.035 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:48.036 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:48.036 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:48.036 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:48.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:48.036 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:48.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:48.037 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:48.037 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:48.037 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:48.037 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.492727_137.1", 2023-10-12T19:31:48.038 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:48.038 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:48.038 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:48.038 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.492727Z", 2023-10-12T19:31:48.039 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:48.039 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:48.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:48.040 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:48.040 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:48.040 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:48.040 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:48.040 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:48.040 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:48.041 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:48.041 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:48.041 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:48.041 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:48.041 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:48.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:48.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:48.042 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:48.042 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:48.043 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:48.043 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:48.043 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:48.043 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:48.043 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:48.043 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:48.044 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:48.044 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:48.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:48.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:48.045 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:48.045 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:48.045 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:48.045 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:48.045 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:48.046 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:48.046 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:48.046 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:48.046 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:48.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:48.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:48.047 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:48.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:48.048 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:48.048 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:48.048 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:48.048 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:48.048 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:48.049 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.492727_137.1 2023-10-12T19:31:53.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:53.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:53.051 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:53.100 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:53.100 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:53.208 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:53.209 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:53.210 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:31:53.211 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:53.211 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:53.211 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:53.211 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:31:53.212 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:53.212 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:53.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:53.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:53.213 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:53.214 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:53.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:53.215 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:53.215 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:53.215 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:53.215 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:53.216 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:53.216 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:53.216 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:53.216 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:53.217 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:53.217 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:53.217 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:53.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:53.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:53.218 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:53.218 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:53.218 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:53.218 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:53.219 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:53.219 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:53.219 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:53.219 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:53.220 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:53.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:53.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:53.221 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:53.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:53.221 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:53.222 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:53.222 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:53.223 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:53.223 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:53.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:31:58.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:58.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:31:58.228 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:31:58.328 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:31:58.328 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:31:58.475 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:31:58.475 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:31:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:31:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:31:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:31:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:31:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:31:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:31:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:31:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:31:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:58.479 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:31:58.479 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:58.479 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:58.479 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:31:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:31:58.481 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:58.481 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:58.481 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:58.481 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:31:58.481 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:31:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:58.483 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:58.483 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:58.483 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:58.483 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:58.483 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:58.484 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:58.484 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:58.484 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:58.484 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:58.484 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:31:58.485 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:31:58.485 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:31:58.485 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:31:58.485 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:31:58.485 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:31:58.486 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:31:58.486 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:31:58.486 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:31:58.486 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:31:58.486 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:58.487 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:58.487 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:31:58.487 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:31:58.487 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:31:58.487 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:31:58.487 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:31:58.488 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:31:58.488 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:31:58.489 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:32:03.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:03.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:03.493 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:03.576 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:03.576 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:03.695 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:03.695 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:03.695 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:03.695 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:03.696 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:03.696 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:03.696 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:03.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:03.696 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:03.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:03.697 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:03.697 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:03.697 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:03.697 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:32:03.698 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:03.698 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:03.698 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:03.698 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:32:03.698 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:03.699 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:03.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:03.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:03.699 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:03.699 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:03.700 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:03.700 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:03.700 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:03.700 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:03.700 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:03.701 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:03.701 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:03.701 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:03.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:03.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:03.702 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:03.702 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:03.702 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:03.702 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:03.702 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:03.703 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:03.703 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:03.703 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:03.703 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:03.703 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:03.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:03.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:03.704 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:03.704 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:03.704 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:03.704 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:03.704 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:03.705 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:03.705 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:03.705 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:03.705 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:03.705 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:03.706 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:03.706 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:03.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:03.706 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:03.706 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:03.707 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:03.707 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:03.707 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:03.708 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.432502_127.1 2023-10-12T19:32:08.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:08.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:08.711 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:08.798 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:08.798 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:08.897 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:08.897 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:08.898 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:08.898 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:08.898 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:08.898 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:08.898 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:08.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:08.899 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:08.899 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:08.899 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:08.899 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:08.900 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:08.900 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:32:08.900 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:08.900 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:08.900 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:08.901 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:32:08.901 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:08.901 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:08.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:08.901 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:08.901 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:08.902 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:08.902 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:08.902 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:08.902 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:08.903 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:08.903 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:08.903 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:08.903 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:08.903 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:08.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:08.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:08.904 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:08.904 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:08.904 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:08.904 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:08.905 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:08.905 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:08.905 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:08.905 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:08.905 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:08.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:08.906 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:08.906 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:08.906 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:08.906 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:08.907 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:08.907 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:08.907 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:08.907 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:08.907 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:08.909 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:08.909 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:08.909 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:08.909 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:08.909 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:08.911 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.435183_128.1 2023-10-12T19:32:13.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:13.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:13.913 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:14.021 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:14.022 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:14.414 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:14.415 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:14.415 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:14.415 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:14.415 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:14.416 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:14.416 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:14.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:14.416 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:14.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:14.417 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:14.417 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:14.417 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:14.417 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:32:14.417 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:14.418 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:14.418 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:14.418 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:32:14.418 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:14.419 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:14.419 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:14.419 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:14.419 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:14.419 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:14.420 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:14.420 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:14.420 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:14.420 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:14.420 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:14.421 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:14.421 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:14.421 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:14.421 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:14.421 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:14.422 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:14.422 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:14.422 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:14.422 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:14.422 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:14.423 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:14.423 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:14.423 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:14.423 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:14.423 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:14.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:14.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:14.424 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:14.424 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:14.424 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:14.425 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:14.425 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:14.425 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:14.425 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:14.425 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:14.425 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:14.426 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:14.426 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:14.426 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:14.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:14.427 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:14.427 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:14.427 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:14.427 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:14.427 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:14.428 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.436071_129.1 2023-10-12T19:32:19.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:19.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:19.431 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:19.508 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:19.508 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:19.873 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:19.873 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:19.874 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:19.874 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:19.874 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:19.874 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:19.874 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:19.875 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:19.875 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:19.875 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:19.875 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:19.875 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:19.876 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:19.876 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:32:19.876 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:19.876 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:19.876 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:19.877 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:32:19.877 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:19.877 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:19.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:19.877 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:19.878 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:19.878 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:19.878 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:19.878 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:19.878 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:19.879 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:19.879 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:19.879 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:19.879 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:19.879 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:19.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:19.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:19.880 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:19.880 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:19.880 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:19.880 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:19.881 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:19.881 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:19.881 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:19.881 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:19.881 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:19.882 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:19.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:19.882 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:19.882 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:19.882 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:19.883 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:19.883 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:19.883 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:19.883 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:19.883 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:19.884 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:19.884 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:19.884 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:19.884 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:19.884 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:19.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:19.885 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:19.885 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:19.885 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:19.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:19.885 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:19.886 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:32:24.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:24.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:24.888 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:24.958 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:24.958 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:25.345 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:25.346 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:25.346 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:25.346 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:25.346 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:25.347 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:25.347 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:25.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:25.347 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:25.347 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:25.348 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:25.348 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:25.348 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:25.348 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:32:25.348 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:25.348 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:25.349 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:25.350 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:32:25.350 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:25.350 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:25.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:25.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:25.351 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:25.351 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:25.351 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:25.351 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:25.351 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:25.352 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:25.352 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:25.352 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:25.352 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:25.352 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:25.353 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:25.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:25.353 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:25.353 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:25.353 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:25.354 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:25.354 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:25.354 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:25.354 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:25.354 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:25.355 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:25.355 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:25.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:25.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:25.356 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:25.356 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:25.356 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:25.356 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:25.356 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:25.357 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:25.357 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:25.357 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:25.357 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:25.357 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:25.357 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:25.358 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:25.358 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:25.358 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:25.358 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:25.359 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:25.359 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:25.359 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:25.360 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:32:30.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:30.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:30.363 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:30.407 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:30.407 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:30.462 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:30.462 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:30.463 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:30.463 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:30.463 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:30.463 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:30.463 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:30.464 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:30.464 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:30.464 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:30.464 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:30.464 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:30.465 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:30.465 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.568332_143.1", 2023-10-12T19:32:30.465 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:30.465 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:30.465 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:30.465 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.568332Z", 2023-10-12T19:32:30.466 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:30.466 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:30.466 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:30.466 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:30.466 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:30.467 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:30.467 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:30.467 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:30.467 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:30.467 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:30.468 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:30.468 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:30.468 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:30.468 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:30.469 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:30.469 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:30.469 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:30.469 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:30.469 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:30.470 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:30.470 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:30.470 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:30.470 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:30.470 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:30.471 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:30.471 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:30.471 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:30.471 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:30.471 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:30.472 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:30.472 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:30.472 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:30.472 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:30.472 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:30.473 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:30.473 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:30.473 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:30.473 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:30.473 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:30.473 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:30.474 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:30.474 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:30.474 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:30.474 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:30.475 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:30.475 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:30.475 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.568332_143.1 2023-10-12T19:32:35.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:35.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:35.478 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:35.520 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:35.520 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:35.572 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:35.572 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:35.572 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:35.573 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:35.573 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:35.573 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:35.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:35.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:35.574 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:35.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:35.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:35.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:35.574 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.568332_143.1", 2023-10-12T19:32:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.568332Z", 2023-10-12T19:32:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:35.576 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:35.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:35.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:35.576 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:35.576 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:35.577 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:35.577 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:35.577 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:35.577 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:35.577 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:35.578 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:35.578 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:35.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:35.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:35.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:35.581 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:35.581 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:35.581 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:35.581 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:35.582 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:35.582 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:35.582 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:35.582 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:35.582 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:35.583 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:35.583 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:35.583 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:35.583 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:35.583 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:35.584 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:35.584 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:35.584 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:35.584 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:35.584 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:35.584 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:35.585 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:35.585 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:35.585 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:35.585 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:35.585 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:35.586 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:35.586 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:35.586 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:35.586 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:35.586 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:35.587 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.568332_143.1 2023-10-12T19:32:40.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:40.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:40.590 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:40.632 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:40.632 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:40.685 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:40.686 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:40.686 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:40.686 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:40.687 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:40.687 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:40.687 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:40.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:40.687 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:40.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:40.688 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:40.688 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:40.688 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:40.688 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:32:40.688 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:40.689 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:40.689 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:40.689 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:32:40.689 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:40.689 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:40.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:40.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:40.691 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:40.692 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:40.692 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:40.692 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:40.692 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:40.692 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:40.693 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:40.693 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:40.693 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:40.693 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:40.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:40.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:40.694 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:40.694 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:40.694 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:40.694 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:40.695 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:40.695 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:40.695 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:40.695 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:40.695 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:40.696 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:40.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:40.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:40.696 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:40.696 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:40.697 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:40.697 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:40.697 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:40.697 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:40.697 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:40.698 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:40.698 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:40.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:40.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:40.699 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:40.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:40.699 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:40.699 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:40.699 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:40.700 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:40.700 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:40.701 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:32:45.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:45.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:45.704 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:45.790 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:45.790 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:45.847 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:45.847 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:45.847 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:45.848 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:45.848 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:45.848 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:45.848 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:45.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:45.849 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:45.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:45.849 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:45.849 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:45.850 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:45.850 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:32:45.850 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:45.850 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:45.851 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:45.851 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:32:45.851 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:45.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:45.852 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:45.852 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:45.852 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:45.852 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:45.852 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:45.853 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:45.853 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:45.853 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:45.853 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:45.853 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:45.854 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:45.854 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:45.854 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:45.854 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:45.854 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:45.855 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:45.855 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:45.855 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:45.855 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:45.856 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:45.856 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:45.856 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:45.857 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:45.857 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:45.857 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:45.858 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:45.858 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:45.858 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:45.858 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:45.858 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:45.859 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:45.859 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:45.859 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:45.859 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:45.859 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:45.860 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:45.860 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:45.860 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:45.860 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:45.860 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:45.860 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:45.861 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:45.861 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:45.861 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:45.862 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.410419_126.1 2023-10-12T19:32:50.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:50.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:50.865 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:50.949 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:50.950 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:51.378 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:51.379 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:51.379 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:51.379 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:51.379 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:51.380 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:51.380 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:51.380 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:51.380 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:51.380 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:51.381 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:51.381 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:51.381 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:51.381 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:32:51.381 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:51.382 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:51.382 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:51.382 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:32:51.382 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:51.382 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:51.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:51.383 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:51.383 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:51.383 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:51.383 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:51.384 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:51.384 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:51.384 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:51.384 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:51.385 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:51.385 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:51.385 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:51.385 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:51.385 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:51.385 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:51.386 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:51.386 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:51.386 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:51.386 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:51.386 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:51.387 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:51.387 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:51.387 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:51.387 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:51.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:51.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:51.388 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:51.388 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:51.388 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:51.388 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:51.389 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:51.389 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:51.389 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:51.389 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:51.389 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:51.390 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:51.390 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:51.390 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:51.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:51.390 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:51.390 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:51.391 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:51.391 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:51.391 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:51.392 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:32:56.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:56.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:32:56.394 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:32:56.486 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:32:56.486 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:32:56.542 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:32:56.543 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:32:56.543 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:32:56.543 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:32:56.543 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:32:56.543 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:32:56.544 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:56.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:56.544 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:32:56.544 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:56.544 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:32:56.545 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:56.545 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:56.545 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:32:56.545 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:56.545 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:32:56.546 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:56.546 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:32:56.546 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:56.546 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:56.546 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:56.547 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:56.547 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:32:56.547 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:56.547 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:56.547 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:32:56.548 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:56.548 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:56.548 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:56.548 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:32:56.548 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:56.549 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:56.549 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:56.549 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:56.549 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:32:56.549 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:56.549 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:56.550 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:56.550 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:56.550 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:56.550 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:56.550 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:56.551 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:56.551 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:56.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:56.551 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:32:56.551 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:32:56.552 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:32:56.552 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:32:56.552 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:32:56.552 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:32:56.552 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:32:56.552 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:32:56.553 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:32:56.553 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:32:56.553 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:56.553 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:56.553 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:32:56.554 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:32:56.554 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:32:56.554 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:32:56.554 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:32:56.554 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:32:56.554 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:32:56.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.470944_133.1 2023-10-12T19:33:01.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:01.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:01.558 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:01.657 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:01.657 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:01.985 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:01.986 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:01.986 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:01.986 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:01.986 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:01.987 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:01.987 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:01.987 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:01.988 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:01.988 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:01.988 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:01.988 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:01.989 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:01.989 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:33:01.989 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:01.989 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:01.990 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:01.990 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:33:01.990 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:01.990 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:01.991 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:01.991 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:01.991 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:01.992 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:01.993 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:01.994 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:01.995 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:01.996 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:01.996 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:01.996 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:01.996 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:01.996 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:01.997 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:01.997 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:01.997 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:01.997 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:01.997 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:01.998 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:01.998 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:01.998 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:01.998 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:01.999 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:01.999 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:01.999 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:01.999 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:01.999 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:02.000 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:02.000 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:02.001 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:33:07.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:07.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:07.003 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:07.065 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:07.066 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:07.506 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:07.507 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:07.507 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:07.507 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:07.507 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:07.508 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:07.508 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:07.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:07.508 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:07.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:07.509 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:07.509 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:07.509 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:07.509 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:33:07.509 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:07.510 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:07.510 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:07.510 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:33:07.510 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:07.511 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:07.511 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:07.511 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:07.511 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:07.511 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:07.511 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:07.512 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:07.512 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:07.512 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:07.512 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:07.512 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:07.513 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:07.513 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:07.513 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:07.513 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:07.513 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:07.514 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:07.514 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:07.514 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:07.514 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:07.514 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:07.515 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:07.515 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:07.515 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:07.515 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:07.515 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:07.516 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:07.516 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:07.516 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:07.516 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:07.516 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:07.516 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:07.517 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:07.517 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:07.517 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:07.517 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:07.517 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:07.518 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:07.518 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:07.518 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:07.518 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:07.519 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:07.519 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:07.519 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:07.519 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:07.520 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.470944_133.1 2023-10-12T19:33:12.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:12.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:12.522 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:12.566 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:12.566 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:12.708 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:12.708 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:12.708 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:12.709 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:12.709 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:12.709 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:12.710 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:12.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:12.710 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:12.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:12.711 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:12.711 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:12.712 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:12.712 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:33:12.712 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:12.713 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:12.713 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:12.713 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:33:12.713 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:12.714 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:12.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:12.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:12.714 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:12.715 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:12.715 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:12.715 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:12.715 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:12.715 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:12.716 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:12.716 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:12.716 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:12.716 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:12.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:12.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:12.717 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:12.717 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:12.717 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:12.718 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:12.718 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:12.718 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:12.718 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:12.718 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:12.719 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:12.719 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:12.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:12.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:12.719 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:12.720 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:12.720 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:12.720 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:12.720 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:12.720 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:12.720 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:12.722 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:12.722 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:12.722 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:12.723 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:12.723 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:12.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:12.723 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:12.723 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:12.724 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:12.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:12.724 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:12.724 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:33:17.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:17.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:17.727 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:17.771 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:17.771 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:17.825 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:17.825 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:17.826 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:17.826 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:17.826 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:17.826 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:17.826 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:17.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:17.827 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:17.827 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:17.827 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:17.827 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:17.828 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:17.828 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:33:17.828 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:17.828 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:17.828 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:17.829 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:33:17.829 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:17.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:17.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:17.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:17.830 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:17.830 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:17.830 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:17.830 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:17.831 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:17.831 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:17.831 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:17.831 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:17.831 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:17.831 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:17.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:17.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:17.832 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:17.832 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:17.832 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:17.833 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:17.833 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:17.833 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:17.833 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:17.833 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:17.834 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:17.834 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:17.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:17.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:17.834 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:17.834 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:17.835 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:17.835 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:17.835 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:17.835 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:17.835 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:17.836 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:17.836 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:17.836 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:17.836 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:17.836 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:17.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:17.837 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:17.837 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:17.837 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:17.837 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:17.837 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:17.838 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:33:22.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:22.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:22.841 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:22.885 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:22.885 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:22.940 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:22.940 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:22.940 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:22.941 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:22.941 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:22.941 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:22.941 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:22.941 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:22.942 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:22.942 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:22.942 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:22.942 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:22.942 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:22.943 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.486122_135.1", 2023-10-12T19:33:22.943 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:22.943 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:22.943 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:22.943 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.486122Z", 2023-10-12T19:33:22.944 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:22.944 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:22.944 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:22.944 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:22.945 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:22.945 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:22.945 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:22.945 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:22.945 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:22.946 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:22.946 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:22.946 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:22.946 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:22.947 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:22.947 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:22.947 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:22.947 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:22.948 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:22.948 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:22.948 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:22.948 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:22.948 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:22.949 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:22.949 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:22.949 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:22.949 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:22.949 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:22.950 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:22.950 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:22.950 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:22.950 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:22.950 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:22.951 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:22.951 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:22.951 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:22.951 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:22.951 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:22.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:22.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:22.952 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:22.952 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:22.952 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:22.953 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:22.953 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:22.953 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:22.953 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:22.957 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.486122_135.1 2023-10-12T19:33:27.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:27.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:27.960 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:28.044 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:28.045 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:28.271 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:28.272 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:28.272 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:28.272 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:28.272 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:28.273 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:28.273 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:28.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:28.273 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:28.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:28.274 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:28.274 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:28.274 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:28.274 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:33:28.274 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:28.275 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:28.275 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:28.275 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:33:28.275 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:28.275 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:28.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:28.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:28.276 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:28.276 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:28.276 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:28.277 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:28.277 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:28.277 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:28.277 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:28.277 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:28.278 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:28.278 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:28.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:28.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:28.278 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:28.279 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:28.279 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:28.279 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:28.279 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:28.279 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:28.280 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:28.280 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:28.280 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:28.280 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:28.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:28.280 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:28.281 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:28.281 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:28.281 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:28.281 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:28.281 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:28.282 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:28.282 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:28.282 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:28.282 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:28.282 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:28.283 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:28.283 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:28.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:28.283 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:28.283 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:28.283 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:28.284 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:28.284 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:28.285 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:33:33.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:33.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:33.288 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:33.334 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:33.334 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:33.397 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:33.397 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:33.397 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:33.397 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:33.397 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:33.398 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:33.398 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:33.398 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:33.398 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:33.398 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:33.399 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:33.399 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:33.399 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:33.399 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:33:33.400 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:33.400 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:33.400 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:33.400 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:33:33.400 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:33.401 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:33.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:33.401 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:33.401 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:33.401 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:33.402 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:33.402 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:33.402 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:33.402 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:33.403 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:33.403 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:33.403 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:33.405 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:33.405 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:33.406 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:33.406 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:33.406 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:33.406 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:33.406 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:33.407 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:33.407 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:33.407 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:33.407 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:33.408 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:33.408 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:33.408 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:33.408 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:33.408 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:33.409 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:33.409 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:33.409 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:33.409 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:33.410 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:33.410 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:33.410 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:33.410 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:33.410 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:33.411 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:33.411 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:33.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:33.411 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:33.411 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:33.412 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:33.412 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:33.412 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:33.413 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:33:38.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:38.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:38.416 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:38.482 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:38.482 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:38.588 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:38.588 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:38.588 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:38.589 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:38.589 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:38.589 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:38.589 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:38.589 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:38.590 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:38.590 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:38.590 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:38.590 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:38.590 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:38.591 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:33:38.591 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:38.591 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:38.591 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:38.591 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:33:38.592 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:38.592 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:38.592 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:38.592 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:38.592 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:38.592 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:38.593 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:38.593 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:38.593 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:38.593 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:38.593 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:38.594 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:38.594 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:38.594 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:38.594 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:38.594 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:38.595 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:38.595 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:38.595 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:38.595 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:38.595 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:38.595 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:38.596 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:38.596 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:38.596 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:38.596 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:38.596 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:38.597 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:38.597 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:38.597 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:38.597 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:38.597 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:38.598 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:38.598 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:38.598 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:38.598 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:38.598 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:38.599 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:38.599 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:38.599 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:38.599 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:38.599 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:38.600 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:38.600 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:38.600 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:38.600 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:38.601 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.436071_129.1 2023-10-12T19:33:43.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:43.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:43.604 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:43.678 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:43.678 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:44.019 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:44.020 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:44.020 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:44.020 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:44.020 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:44.021 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:44.021 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:44.021 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:44.021 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:44.021 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:44.022 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:44.022 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:44.022 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:44.022 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:33:44.022 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:44.023 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:44.023 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:44.023 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:33:44.023 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:44.023 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:44.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:44.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:44.024 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:44.024 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:44.024 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:44.025 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:44.025 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:44.025 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:44.025 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:44.025 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:44.026 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:44.026 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:44.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:44.026 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:44.026 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:44.027 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:44.027 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:44.027 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:44.027 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:44.027 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:44.028 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:44.028 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:44.028 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:44.028 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:44.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:44.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:44.029 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:44.029 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:44.029 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:44.029 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:44.029 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:44.030 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:44.030 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:44.030 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:44.030 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:44.030 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:44.031 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:44.031 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:44.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:44.031 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:44.031 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:44.032 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:44.032 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:44.032 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:44.033 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:33:49.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:49.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:49.035 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:49.121 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:49.122 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:49.176 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:49.176 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:49.176 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:49.177 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:49.177 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:49.177 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:49.177 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:49.177 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:49.178 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:49.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:49.178 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:49.178 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:49.178 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:49.179 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.589196_150.1", 2023-10-12T19:33:49.179 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:49.179 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:49.179 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:49.180 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.589196Z", 2023-10-12T19:33:49.180 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:49.180 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:49.180 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:49.180 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:49.181 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:49.181 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:49.181 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:49.181 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:49.181 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:49.182 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:49.182 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:49.182 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:49.182 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:49.182 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:49.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:49.183 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:49.183 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:49.183 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:49.183 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:49.184 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:49.184 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:49.184 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:49.184 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:49.184 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:49.185 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:49.185 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:49.185 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:49.185 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:49.185 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:49.186 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:49.186 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:49.186 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:49.186 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:49.187 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:49.187 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:49.187 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:49.187 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:49.187 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:49.188 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:49.188 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:49.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:49.188 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:49.189 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:49.189 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:49.189 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:49.189 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:49.190 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.589196_150.1 2023-10-12T19:33:54.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:54.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:54.193 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:54.274 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:54.275 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:54.668 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:54.668 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:54.669 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:54.669 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:54.669 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:54.669 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:54.669 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:54.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:54.670 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:54.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:54.670 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:54.671 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:54.671 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:54.671 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.589196_150.1", 2023-10-12T19:33:54.672 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:54.672 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:54.672 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:54.673 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.589196Z", 2023-10-12T19:33:54.673 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:54.673 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:54.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:54.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:54.674 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:54.675 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:54.675 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:54.675 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:54.676 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:54.676 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:54.676 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:54.677 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:54.677 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:54.677 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:54.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:54.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:54.678 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:54.678 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:54.679 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:54.679 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:54.679 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:54.679 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:54.679 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:54.680 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:54.680 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:54.680 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:54.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:54.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:54.681 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:54.681 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:54.681 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:54.681 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:54.681 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:54.682 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:54.682 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:54.682 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:54.682 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:54.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:54.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:54.683 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:54.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:54.683 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:54.684 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:54.684 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:54.684 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:54.684 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:54.685 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.589196_150.1 2023-10-12T19:33:59.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:59.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:33:59.689 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:33:59.738 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:33:59.738 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:33:59.798 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:33:59.798 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:33:59.798 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:33:59.799 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:33:59.799 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:33:59.799 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:33:59.799 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:59.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:59.800 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:33:59.800 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:59.800 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:33:59.800 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:59.801 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:59.801 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T19:33:59.801 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:59.802 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:33:59.802 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:59.802 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T19:33:59.802 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:59.802 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:59.803 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:59.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:59.803 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:33:59.803 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:59.804 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:59.804 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:33:59.804 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:59.804 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:59.804 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:59.805 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:33:59.805 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:59.805 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:59.805 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:59.805 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:59.806 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:33:59.806 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:59.808 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:59.808 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:59.809 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:59.809 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:59.809 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:59.809 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:59.809 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:59.810 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:59.810 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:59.810 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:33:59.810 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:33:59.810 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:33:59.811 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:33:59.811 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:33:59.811 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:33:59.811 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:33:59.811 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:33:59.812 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:33:59.812 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:33:59.812 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:59.812 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:59.812 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:33:59.812 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:33:59.813 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:33:59.813 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:33:59.813 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:33:59.813 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:33:59.813 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:33:59.815 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.473047_134.1 2023-10-12T19:34:04.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:04.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:04.816 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:04.858 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:04.859 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:04.911 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:04.911 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:04.911 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:04.912 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:04.912 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:04.912 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:04.912 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:04.913 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:04.913 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:04.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:04.913 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:04.914 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:04.914 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:04.914 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.545187_139.1", 2023-10-12T19:34:04.915 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:04.915 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:04.915 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:04.916 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.545187Z", 2023-10-12T19:34:04.916 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:04.916 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:04.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:04.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:04.918 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:04.918 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:04.918 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:04.919 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:04.919 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:04.919 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:04.920 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:04.920 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:04.921 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:04.921 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:04.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:04.922 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:04.922 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:04.923 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:04.923 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:04.923 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:04.923 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:04.924 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:04.924 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:04.924 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:04.925 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:04.925 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:04.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:04.925 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:04.926 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:04.926 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:04.926 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:04.926 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:04.926 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:04.927 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:04.927 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:04.927 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:04.927 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:04.927 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:04.927 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:04.928 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:04.928 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:04.928 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:04.928 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:04.928 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:04.929 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:04.929 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:04.930 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.545187_139.1 2023-10-12T19:34:09.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:09.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:09.932 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:09.974 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:09.975 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:10.029 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:10.030 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:10.030 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:10.030 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:10.030 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:10.030 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:10.031 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:10.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:10.031 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:10.031 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:10.031 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:10.032 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:10.032 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:10.032 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T19:34:10.032 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:10.032 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:10.033 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:10.033 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T19:34:10.033 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:10.033 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:10.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:10.034 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:10.034 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:10.034 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:10.034 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:10.034 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:10.035 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:10.035 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:10.035 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:10.035 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:10.035 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:10.035 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:10.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:10.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:10.036 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:10.036 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:10.036 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:10.037 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:10.037 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:10.037 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:10.037 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:10.037 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:10.037 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:10.038 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:10.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:10.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:10.038 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:10.039 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:10.039 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:10.039 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:10.039 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:10.039 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:10.039 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:10.040 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:10.040 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:10.040 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:10.040 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:10.040 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:10.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:10.041 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:10.041 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:10.041 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:10.041 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:10.041 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:10.043 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.438188_130.1 2023-10-12T19:34:15.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:15.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:15.045 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:15.088 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:15.088 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:15.139 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:15.140 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:15.140 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:15.140 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:15.140 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:15.141 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:15.141 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:15.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:15.141 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:15.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:15.142 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:15.142 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:15.142 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:15.143 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:34:15.143 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:15.143 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:15.143 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:15.143 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:34:15.144 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:15.144 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:15.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:15.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:15.144 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:15.145 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:15.145 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:15.145 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:15.145 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:15.146 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:15.146 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:15.146 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:15.146 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:15.146 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:15.147 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:15.147 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:15.147 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:15.147 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:15.147 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:15.148 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:15.148 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:15.148 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:15.148 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:15.148 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:15.149 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:15.149 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:15.149 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:15.149 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:15.149 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:15.150 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:15.150 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:15.150 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:15.150 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:15.150 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:15.151 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:15.151 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:15.151 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:15.151 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:15.152 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:15.152 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:15.152 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:15.152 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:15.152 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:15.153 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:15.153 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:15.153 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:15.155 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:34:20.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:20.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:20.157 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:20.265 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:20.266 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:20.420 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:20.420 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:20.421 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:20.421 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:20.421 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:20.421 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:20.421 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:20.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:20.422 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:20.422 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:20.422 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:20.422 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:20.423 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:20.423 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:34:20.423 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:20.423 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:20.423 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:20.424 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:34:20.424 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:20.424 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:20.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:20.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:20.425 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:20.425 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:20.425 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:20.425 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:20.425 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:20.426 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:20.426 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:20.426 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:20.426 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:20.426 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:20.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:20.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:20.427 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:20.427 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:20.427 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:20.428 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:20.428 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:20.428 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:20.428 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:20.428 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:20.428 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:20.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:20.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:20.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:20.430 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:20.430 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:20.431 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:20.432 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:20.432 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:20.433 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:20.434 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:20.434 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:20.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:20.434 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:20.435 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:20.437 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.407338_124.1 2023-10-12T19:34:25.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:25.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:25.439 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:25.487 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:25.488 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:25.586 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:25.586 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:25.586 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:25.587 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:25.587 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:25.587 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:25.587 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:25.587 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:25.588 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:25.588 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:25.588 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:25.588 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:25.588 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:25.588 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:34:25.589 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:25.589 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:25.589 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:25.589 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:34:25.589 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:25.590 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:25.590 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:25.590 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:25.590 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:25.590 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:25.591 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:25.591 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:25.591 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:25.591 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:25.591 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:25.592 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:25.592 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:25.592 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:25.592 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:25.592 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:25.593 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:25.593 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:25.593 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:25.593 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:25.593 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:25.594 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:25.594 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:25.594 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:25.594 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:25.595 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:25.595 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:25.595 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:25.595 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:25.595 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:25.595 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:25.596 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:25.596 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:25.596 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:25.596 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:25.596 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:25.597 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:25.597 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:25.597 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:25.597 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:25.597 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:25.597 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:25.598 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:25.598 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:25.598 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:25.598 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:25.599 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:34:30.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:30.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:30.601 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:30.718 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:30.718 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:30.894 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:30.895 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:30.895 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:30.895 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:30.895 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:30.896 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:30.896 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:30.896 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:30.896 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:30.896 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:30.897 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:30.897 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:30.897 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:30.897 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:34:30.897 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:30.898 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:30.898 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:30.898 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:34:30.898 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:30.899 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:30.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:30.899 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:30.899 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:30.899 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:30.900 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:30.900 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:30.900 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:30.900 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:30.900 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:30.901 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:30.901 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:30.901 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:30.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:30.901 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:30.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:30.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:30.904 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:30.904 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:30.905 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:30.905 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:30.905 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:30.905 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:30.905 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:30.906 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:30.906 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:30.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:30.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:30.906 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:30.907 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:30.907 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:30.907 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:30.907 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:30.908 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:30.908 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:30.909 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:34:35.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:35.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:35.912 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:35.994 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:35.995 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:36.433 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:36.434 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:36.434 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:36.434 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:36.434 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:36.435 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:36.435 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:36.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:36.435 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:36.435 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:36.436 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:36.436 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:36.436 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:36.436 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.492727_137.1", 2023-10-12T19:34:36.436 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:36.437 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:36.437 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:36.437 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.492727Z", 2023-10-12T19:34:36.437 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:36.437 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:36.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:36.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:36.438 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:36.438 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:36.438 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:36.439 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:36.439 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:36.439 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:36.439 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:36.439 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:36.440 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:36.440 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:36.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:36.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:36.440 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:36.440 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:36.441 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:36.441 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:36.441 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:36.441 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:36.441 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:36.442 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:36.442 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:36.442 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:36.442 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:36.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:36.443 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:36.443 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:36.443 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:36.443 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:36.443 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:36.443 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:36.444 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:36.444 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:36.444 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:36.444 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:36.444 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:36.445 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:36.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:36.445 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:36.445 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:36.445 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:36.445 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:36.446 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:36.446 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.492727_137.1 2023-10-12T19:34:41.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:41.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:41.449 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:41.491 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:41.491 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:41.544 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:41.544 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:41.544 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:41.544 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:41.545 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:41.545 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:41.545 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:41.545 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:41.545 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:41.546 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:41.546 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:41.546 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:41.546 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:41.546 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:34:41.547 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:41.547 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:41.547 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:41.547 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:34:41.547 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:41.548 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:41.548 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:41.548 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:41.548 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:41.548 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:41.549 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:41.549 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:41.549 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:41.549 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:41.549 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:41.550 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:41.550 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:41.550 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:41.550 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:41.550 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:41.550 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:41.551 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:41.551 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:41.551 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:41.551 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:41.552 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:41.552 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:41.552 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:41.552 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:41.553 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:41.553 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:41.553 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:41.553 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:41.553 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:41.553 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:41.554 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:41.554 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:41.554 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:41.554 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:41.554 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:41.555 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:41.555 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:41.555 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:41.555 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:41.555 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:41.555 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:41.556 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:41.556 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:41.556 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:41.556 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:41.557 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.436071_129.1 2023-10-12T19:34:46.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:46.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:46.560 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:46.603 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:46.604 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.545187_139.1", 2023-10-12T19:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.545187Z", 2023-10-12T19:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:46.908 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.545187_139.1 2023-10-12T19:34:51.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:51.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:51.910 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:51.960 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:51.960 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:52.014 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:52.014 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:52.015 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:52.015 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:52.015 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:52.015 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:52.015 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:52.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:52.016 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:52.016 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:52.016 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:52.016 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:52.017 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:52.017 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:34:52.017 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:52.017 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:52.017 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:52.018 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:34:52.018 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:52.018 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:52.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:52.018 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:52.019 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:52.019 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:52.019 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:52.019 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:52.019 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:52.020 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:52.020 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:52.020 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:52.020 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:52.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:52.021 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:52.021 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:52.021 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:52.021 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:52.021 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:52.022 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:52.022 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:52.022 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:52.022 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:52.022 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:52.022 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:52.023 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:52.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:52.023 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:52.023 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:52.023 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:52.024 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:52.024 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:52.024 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:52.024 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:52.024 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:52.025 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:52.025 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:52.025 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:52.025 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:52.025 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:52.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:52.026 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:52.026 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:52.026 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:52.026 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:52.026 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:52.028 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.470944_133.1 2023-10-12T19:34:57.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:57.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:34:57.030 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:34:57.073 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:34:57.073 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:34:57.125 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:34:57.126 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:34:57.126 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:34:57.126 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:34:57.126 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:34:57.127 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:34:57.127 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:57.127 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:57.127 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:34:57.127 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:57.128 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:34:57.128 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:57.128 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:57.128 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:34:57.128 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:57.129 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:34:57.129 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:57.129 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:34:57.129 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:57.129 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:57.130 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:57.130 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:57.130 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:34:57.130 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:57.131 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:57.131 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:34:57.131 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:57.131 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:57.132 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:57.132 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:34:57.132 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:57.132 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:57.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:57.133 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:57.133 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:34:57.133 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:57.133 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:57.133 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:57.133 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:57.134 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:57.134 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:57.134 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:57.134 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:57.134 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:57.135 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:57.135 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:34:57.135 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:34:57.135 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:34:57.135 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:34:57.136 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:34:57.136 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:34:57.136 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:34:57.136 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:34:57.136 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:34:57.137 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:34:57.137 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:57.137 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:57.137 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:34:57.137 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:34:57.137 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:34:57.138 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:34:57.138 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:34:57.138 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:34:57.138 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:34:57.139 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:35:02.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:02.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:02.142 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:02.226 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:02.226 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:02.306 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:02.306 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:02.307 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:02.307 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:02.307 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:02.307 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:02.307 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:02.308 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:02.309 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:02.309 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T19:35:02.309 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:02.309 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:02.309 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:02.310 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T19:35:02.310 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:02.310 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:02.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:02.311 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:02.312 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:02.312 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:02.312 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:02.312 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:02.312 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:02.313 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:02.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:02.313 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:02.313 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:02.313 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:02.314 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:02.314 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:02.314 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:02.314 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:02.314 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:02.314 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:02.315 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:02.315 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:02.315 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:02.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:02.316 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:02.316 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:02.316 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:02.316 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:02.316 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:02.317 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:02.317 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:02.317 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:02.317 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:02.317 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:02.318 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:02.318 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:02.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:02.318 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:02.318 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:02.319 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:02.319 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:02.319 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:02.320 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.473047_134.1 2023-10-12T19:35:07.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:07.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:07.323 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:07.398 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:07.399 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:07.569 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:07.569 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:07.569 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:07.570 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:07.570 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:07.570 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:07.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:07.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:07.571 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:07.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:07.572 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:07.572 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:07.572 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:07.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:35:07.572 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:07.573 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:07.573 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:07.573 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:35:07.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:07.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:07.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:07.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:07.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:07.575 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:07.575 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:07.575 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:07.575 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:07.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:07.576 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:07.576 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:07.576 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:07.577 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:07.577 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:07.577 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:07.578 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:07.578 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:07.579 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:07.579 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:07.579 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:07.580 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:07.580 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:07.580 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:07.581 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:07.581 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:07.581 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:07.582 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:07.582 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:07.582 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:07.583 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:07.583 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:07.583 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:07.583 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:07.584 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:07.584 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:07.584 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:07.585 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:07.585 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:07.585 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:07.586 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:07.586 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:07.586 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:07.587 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:07.587 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:07.588 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:07.589 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:35:12.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:12.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:12.593 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:12.680 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:12.680 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:12.801 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:12.801 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:12.801 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:12.801 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:12.802 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:12.802 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:12.802 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:12.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:12.802 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:12.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:12.803 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:12.803 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:12.803 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:12.803 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:35:12.803 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:12.804 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:12.804 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:12.804 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:35:12.804 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:12.804 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:12.805 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:12.805 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:12.805 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:12.805 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:12.805 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:12.806 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:12.806 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:12.806 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:12.806 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:12.806 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:12.807 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:12.807 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:12.807 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:12.807 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:12.807 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:12.807 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:12.808 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:12.808 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:12.808 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:12.808 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:12.808 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:12.809 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:12.809 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:12.809 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:12.809 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:12.809 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:12.809 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:12.810 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:12.810 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:12.810 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:12.810 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:12.810 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:12.811 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:12.811 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:12.811 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:12.811 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:12.811 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:12.812 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:12.812 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:12.812 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:12.812 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:12.812 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:12.812 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:12.813 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:12.814 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139003.336876_324.1 target=1_1697136751.408247_125.1 2023-10-12T19:35:17.816 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... FAIL 2023-10-12T19:35:17.817 INFO:rgw_multi.tests:create bucket zone=a1 name=pqrzfz-53 2023-10-12T19:35:17.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:17.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:17.917 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:17.991 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:17.992 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:18.299 INFO:teuthology.orchestra.run.smithi170.stderr:No --period given, using current period=4fe09484-838f-4904-82b9-13168a9d62ca 2023-10-12T19:35:18.392 INFO:teuthology.orchestra.run.smithi170.stdout:[ 2023-10-12T19:35:18.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:18.393 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697139317.832704_330.1", 2023-10-12T19:35:18.393 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T19:35:17.832704Z" 2023-10-12T19:35:18.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:18.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:18.394 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:18.394 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "2023-10-12T18:50:33.545339Z" 2023-10-12T19:35:18.394 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:18.394 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:18.394 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:18.395 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:35:18.395 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:18.395 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:18.395 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:18.395 INFO:teuthology.orchestra.run.smithi170.stdout: "last_update": "0.000000" 2023-10-12T19:35:18.396 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:18.396 INFO:teuthology.orchestra.run.smithi170.stdout:] 2023-10-12T19:35:18.397 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T19:35:18.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:18.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:18.398 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:18.445 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:18.445 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:18.498 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:18.498 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:18.499 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:18.499 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:18.499 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:18.499 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:18.499 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:18.500 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:18.500 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:18.500 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:18.500 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:18.500 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:18.500 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:18.501 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:35:18.501 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:18.501 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:18.501 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:18.501 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:35:18.502 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:18.502 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:18.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:18.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:18.503 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:18.503 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:18.503 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:18.503 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:18.503 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:18.504 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:18.504 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:18.504 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:18.504 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:18.504 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:18.505 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:18.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:18.505 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:18.505 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:18.505 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:18.506 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:18.506 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:18.507 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:18.507 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:18.507 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:18.507 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:18.507 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:18.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:18.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:18.508 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:18.508 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:18.508 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:18.509 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:18.509 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:18.509 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:18.509 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:18.509 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:18.510 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:18.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:18.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:18.510 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:18.510 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:18.511 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:18.511 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:18.511 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:18.511 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:18.511 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:18.512 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:35:23.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:23.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:23.517 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:23.561 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:23.562 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:23.616 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:23.617 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:23.617 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:23.618 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:23.618 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:23.618 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:23.618 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:23.619 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:23.619 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:23.619 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:23.619 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:23.620 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:23.620 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:23.620 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.486122_135.1", 2023-10-12T19:35:23.620 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:23.621 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:23.621 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:23.621 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.486122Z", 2023-10-12T19:35:23.621 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:23.621 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:23.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:23.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:23.622 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:23.622 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:23.622 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:23.623 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:23.623 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:23.623 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:23.623 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:23.623 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:23.624 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:23.624 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:23.624 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:23.624 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:23.625 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:23.625 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:23.625 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:23.625 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:23.625 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:23.626 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:23.626 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:23.626 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:23.626 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:23.626 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:23.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:23.627 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:23.627 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:23.627 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:23.627 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:23.628 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:23.628 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:23.628 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:23.628 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:23.628 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:23.629 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:23.629 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:23.629 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:23.629 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:23.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:23.629 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:23.630 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:23.630 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:23.630 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:23.630 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:23.631 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.486122_135.1 2023-10-12T19:35:28.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:28.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:28.634 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:28.676 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:28.676 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:28.727 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:28.728 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:28.728 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:28.728 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:28.728 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:28.729 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:28.729 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:28.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:28.729 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:28.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:28.729 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:28.730 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:28.730 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:28.730 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:35:28.730 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:28.731 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:28.731 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:28.731 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:35:28.731 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:28.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:28.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:28.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:28.732 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:28.732 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:28.732 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:28.732 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:28.733 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:28.733 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:28.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:28.733 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:28.733 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:28.733 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:28.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:28.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:28.734 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:28.734 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:28.734 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:28.735 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:28.735 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:28.735 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:28.735 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:28.735 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:28.736 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:28.736 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:28.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:28.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:28.736 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:28.736 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:28.737 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:28.737 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:28.737 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:28.737 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:28.737 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:28.738 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:28.738 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:28.738 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:28.738 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:28.738 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:28.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:28.739 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:28.739 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:28.739 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:28.739 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:28.739 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:28.740 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:35:33.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:33.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:33.743 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:33.785 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:33.786 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:33.839 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:33.840 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:33.840 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:33.840 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:33.840 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:33.841 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:33.841 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:33.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:33.842 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:33.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:33.842 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:33.842 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:35:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:33.843 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:33.844 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:33.844 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:35:33.844 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:33.845 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:33.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:33.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:33.845 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:33.846 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:33.846 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:33.846 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:33.846 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:33.847 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:33.847 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:33.847 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:33.848 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:33.848 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:33.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:33.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:33.849 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:33.849 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:33.849 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:33.849 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:33.850 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:33.850 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:33.850 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:33.851 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:33.851 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:33.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:33.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:33.852 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:33.852 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:33.852 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:33.852 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:33.853 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:33.853 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:33.854 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:33.854 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:33.854 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:33.855 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:33.855 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:33.855 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:33.856 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:33.856 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:33.856 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:33.857 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:33.857 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:33.857 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:33.857 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:33.859 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.436071_129.1 2023-10-12T19:35:38.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:38.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:38.861 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:38.928 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:38.928 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:38.997 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:38.998 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:38.998 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:38.998 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:38.998 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:38.999 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:38.999 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:38.999 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:38.999 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:38.999 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:39.000 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:39.000 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:39.000 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:39.000 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.545187_139.1", 2023-10-12T19:35:39.000 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:39.001 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:39.001 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:39.001 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.545187Z", 2023-10-12T19:35:39.001 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:39.001 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:39.001 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:39.002 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:39.002 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:39.002 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:39.002 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:39.002 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:39.003 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:39.003 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:39.003 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:39.003 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:39.003 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:39.004 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:39.004 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:39.004 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:39.004 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:39.004 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:39.004 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:39.005 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:39.005 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:39.005 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:39.005 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:39.005 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:39.006 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:39.006 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:39.006 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:39.006 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:39.006 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:39.007 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:39.007 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:39.007 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:39.007 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:39.007 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:39.008 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:39.008 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:39.008 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:39.008 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:39.008 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:39.008 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:39.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:39.009 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:39.009 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:39.009 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:39.009 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:39.010 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:39.010 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.545187_139.1 2023-10-12T19:35:44.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:44.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:44.012 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:44.081 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:44.082 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:44.405 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:44.405 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:44.406 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:44.406 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:44.406 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:44.406 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:44.406 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:44.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:44.407 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:44.407 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:44.407 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:44.407 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:44.408 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:44.408 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470052_132.1", 2023-10-12T19:35:44.408 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:44.408 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:44.408 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:44.409 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470052Z", 2023-10-12T19:35:44.409 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:44.409 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:44.409 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:44.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:44.410 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:44.410 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:44.410 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:44.410 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:44.410 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:44.411 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:44.411 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:44.411 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:44.411 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:44.411 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:44.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:44.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:44.412 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:44.412 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:44.412 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:44.413 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:44.413 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:44.413 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:44.413 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:44.413 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:44.414 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:44.414 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:44.414 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:44.414 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:44.414 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:44.414 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:44.415 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:44.415 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:44.415 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:44.415 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:44.415 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:44.416 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:44.416 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:44.416 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:44.416 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:44.416 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:44.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:44.417 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:44.417 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:44.417 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:44.417 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:44.418 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:44.419 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.470052_132.1 2023-10-12T19:35:49.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:49.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:49.422 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:49.489 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:49.490 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:49.912 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:49.941 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:49.941 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:49.942 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:49.942 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:49.942 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:49.942 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:49.943 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:49.943 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:49.943 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:49.943 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:49.943 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:49.944 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:49.944 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.492727_137.1", 2023-10-12T19:35:49.944 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:49.944 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:49.944 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:49.944 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.492727Z", 2023-10-12T19:35:49.945 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:49.945 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:49.945 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:49.945 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:49.946 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:49.946 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:49.946 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:49.946 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:49.947 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:49.947 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:49.947 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:49.947 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:49.948 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:49.948 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:49.948 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:49.948 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:49.948 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:49.949 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:49.949 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:49.949 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:49.949 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:49.949 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:49.949 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:49.950 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:49.950 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:49.950 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:49.950 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:49.951 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:49.951 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:49.951 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:49.951 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:49.951 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:49.952 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:49.952 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:49.952 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:49.952 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:49.953 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:49.953 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:49.953 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:49.953 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:49.953 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:49.954 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:49.954 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:49.954 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:49.954 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:49.954 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:49.955 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.492727_137.1 2023-10-12T19:35:54.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:54.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:35:54.958 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:35:55.029 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:35:55.029 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:35:55.418 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:35:55.418 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:35:55.418 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:35:55.419 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:35:55.419 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:35:55.419 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:35:55.419 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:55.419 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:55.420 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:35:55.420 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:55.420 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:35:55.420 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:55.420 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:55.421 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.610069_152.1", 2023-10-12T19:35:55.421 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:55.421 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:35:55.421 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:55.422 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.610069Z", 2023-10-12T19:35:55.422 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:55.422 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:55.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:55.423 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:55.423 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:35:55.423 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:55.423 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:55.424 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:35:55.424 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:55.424 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:55.424 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:55.424 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:35:55.425 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:55.425 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:55.425 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:55.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:55.425 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:35:55.425 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:55.426 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:55.426 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:55.426 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:55.426 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:55.426 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:55.427 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:55.427 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:55.427 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:55.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:55.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:35:55.428 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:35:55.428 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:35:55.428 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:35:55.428 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:35:55.429 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:35:55.429 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:35:55.429 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:35:55.429 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:35:55.429 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:35:55.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:55.430 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:55.430 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:35:55.430 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:35:55.430 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:35:55.430 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:35:55.431 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:35:55.431 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:35:55.431 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:35:55.433 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.610069_152.1 2023-10-12T19:36:00.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:00.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:00.436 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:00.490 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:00.491 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:00.552 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:00.552 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:00.552 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:00.552 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:00.553 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:00.553 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:00.553 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:00.553 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:00.553 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:00.554 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:00.554 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:00.554 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:00.554 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:00.555 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:36:00.555 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:00.555 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:00.555 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:00.555 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:36:00.556 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:00.556 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:00.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:00.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:00.556 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:00.556 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:00.557 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:00.557 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:00.557 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:00.557 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:00.557 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:00.558 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:00.558 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:00.558 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:00.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:00.559 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:00.559 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:00.559 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:00.559 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:00.559 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:00.560 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:00.560 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:00.560 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:00.560 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:00.560 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:00.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:00.561 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:00.561 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:00.561 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:00.561 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:00.561 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:00.562 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:00.562 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:00.562 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:00.562 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:00.562 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:00.563 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:00.563 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:00.563 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:00.563 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:00.563 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:00.564 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:00.564 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:00.564 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:00.564 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:00.565 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:00.566 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.407338_124.1 2023-10-12T19:36:05.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:05.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:05.571 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:05.666 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:05.666 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:06.003 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:06.003 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:06.004 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:06.004 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:06.004 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:06.004 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:06.004 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:06.005 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:06.005 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:06.005 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:06.005 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:06.005 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:06.006 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:06.006 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.486122_135.1", 2023-10-12T19:36:06.006 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:06.006 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:06.006 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:06.007 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.486122Z", 2023-10-12T19:36:06.007 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:06.007 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:06.007 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:06.007 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:06.008 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:06.008 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:06.008 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:06.008 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:06.008 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:06.008 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:06.009 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:06.009 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:06.009 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:06.009 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:06.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:06.010 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:06.010 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:06.010 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:06.010 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:06.010 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:06.011 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:06.011 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:06.011 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:06.011 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:06.011 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:06.012 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:06.012 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:06.012 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:06.013 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:06.014 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:06.014 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:06.014 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:06.014 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:06.015 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:06.015 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:06.015 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:06.015 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:06.015 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:06.017 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:06.018 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.486122_135.1 2023-10-12T19:36:11.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:11.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:11.021 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:11.089 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:11.090 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:11.506 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:11.506 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:11.506 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:11.507 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:11.507 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:11.507 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:11.507 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:11.509 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:11.509 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:11.509 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:11.510 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:36:11.510 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:11.510 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:11.511 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:11.511 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:36:11.511 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:11.512 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:11.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:11.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:11.513 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:11.513 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:11.513 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:11.514 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:11.514 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:11.514 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:11.515 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:11.515 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:11.516 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:11.516 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:11.516 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:11.517 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:11.517 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:11.517 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:11.518 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:11.518 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:11.518 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:11.518 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:11.519 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:11.519 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:11.519 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:11.519 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:11.519 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:11.520 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:11.520 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:11.520 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:11.520 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:11.520 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:11.520 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:11.521 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:11.521 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:11.521 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:11.521 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:11.521 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:11.522 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:11.522 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:11.522 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:11.522 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:11.522 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:11.525 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:11.525 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:11.526 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:11.527 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:36:16.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:16.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:16.530 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:16.573 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:16.573 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:16.628 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:16.628 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:16.628 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:16.629 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:16.629 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:16.629 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:16.629 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:16.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:16.630 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:16.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:16.630 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:16.630 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:16.630 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:16.631 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:36:16.631 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:16.631 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:16.631 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:16.631 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:36:16.632 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:16.632 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:16.632 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:16.632 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:16.632 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:16.632 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:16.633 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:16.633 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:16.633 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:16.633 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:16.633 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:16.634 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:16.634 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:16.634 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:16.634 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:16.634 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:16.635 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:16.635 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:16.635 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:16.635 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:16.635 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:16.636 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:16.637 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:16.638 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:16.639 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:16.640 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:16.640 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:16.640 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:16.640 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:16.641 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.436071_129.1 2023-10-12T19:36:21.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:21.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:21.643 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:21.687 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:21.687 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:21.740 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:21.741 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:21.741 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:21.741 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:21.741 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:21.741 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:21.742 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:21.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:21.742 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:21.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:21.742 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:21.743 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:21.743 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:21.743 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:36:21.743 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:21.743 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:21.744 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:21.744 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:36:21.744 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:21.744 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:21.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:21.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:21.745 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:21.745 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:21.745 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:21.745 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:21.746 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:21.746 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:21.746 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:21.746 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:21.746 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:21.747 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:21.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:21.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:21.747 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:21.747 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:21.748 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:21.748 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:21.748 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:21.748 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:21.748 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:21.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:21.750 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:21.751 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:21.752 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:21.753 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:21.753 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:21.753 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:21.754 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.470944_133.1 2023-10-12T19:36:26.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:26.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:26.757 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:26.820 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:26.821 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:26.871 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:26.872 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:26.872 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:26.872 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:26.872 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:26.872 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:26.873 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:26.873 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:26.873 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:26.873 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:26.873 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:26.874 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:26.874 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:26.874 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:36:26.874 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:26.874 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:26.875 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:26.875 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:36:26.875 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:26.875 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:26.875 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:26.875 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:26.876 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:26.876 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:26.876 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:26.876 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:26.876 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:26.877 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:26.877 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:26.877 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:26.877 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:26.877 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:26.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:26.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:26.878 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:26.878 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:26.879 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:26.879 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:26.879 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:26.879 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:26.879 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:26.880 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:26.880 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:26.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:26.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:26.881 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:26.881 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:26.881 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:26.881 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:26.881 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:26.882 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:26.882 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:26.882 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:26.882 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:26.883 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:26.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:26.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:26.883 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:26.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:26.884 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:26.884 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:26.884 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:26.884 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:26.884 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:26.885 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.407338_124.1 2023-10-12T19:36:31.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:31.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:31.888 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:31.961 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:31.961 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:32.029 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:32.029 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:32.030 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:32.030 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:32.030 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:32.030 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:32.031 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:32.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:32.031 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:32.031 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:32.031 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:32.031 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:32.032 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:32.032 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:36:32.032 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:32.032 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:32.032 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:32.033 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:36:32.033 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:32.033 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:32.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:32.034 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:32.034 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:32.034 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:32.034 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:32.034 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:32.035 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:32.035 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:32.035 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:32.035 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:32.035 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:32.036 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:32.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:32.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:32.036 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:32.036 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:32.037 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:32.037 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:32.038 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:32.038 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:32.038 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:32.038 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:32.039 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:32.039 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:32.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:32.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:32.040 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:32.040 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:32.040 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:32.040 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:32.040 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:32.041 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:32.041 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:32.041 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:32.041 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:32.041 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:32.042 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:32.042 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:32.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:32.042 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:32.042 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:32.043 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:32.043 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:32.043 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:32.044 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:36:37.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:37.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:37.047 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:37.089 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:37.090 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:37.189 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:37.189 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:37.190 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:37.190 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:37.190 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:37.190 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:37.191 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:37.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:37.191 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:37.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:37.191 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:37.192 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:37.192 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:37.192 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:36:37.192 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:37.192 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:37.193 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:37.193 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:36:37.193 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:37.193 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:37.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:37.194 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:37.194 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:37.194 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:37.194 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:37.194 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:37.195 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:37.195 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:37.195 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:37.195 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:37.196 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:37.196 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:37.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:37.196 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:37.196 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:37.197 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:37.197 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:37.197 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:37.198 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:37.198 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:37.198 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:37.199 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:37.199 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:37.199 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:37.200 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:37.200 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:37.200 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:37.200 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:37.200 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:37.201 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:37.201 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:37.201 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:37.201 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:37.202 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:37.202 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:37.202 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:37.202 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:37.202 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:37.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:37.203 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:37.203 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:37.203 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:37.203 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:37.204 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:37.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:36:42.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:42.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:42.208 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:42.290 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:42.290 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:42.581 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:42.582 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:42.582 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:42.583 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:42.583 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:42.583 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:42.583 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:42.584 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:42.584 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:42.584 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:42.584 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:42.585 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:42.585 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:42.585 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:36:42.586 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:42.586 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:42.587 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:42.587 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:36:42.587 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:42.588 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:42.588 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:42.589 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:42.589 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:42.589 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:42.590 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:42.590 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:42.590 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:42.590 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:42.591 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:42.591 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:42.591 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:42.592 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:42.592 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:42.592 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:42.592 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:42.593 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:42.593 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:42.593 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:42.594 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:42.594 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:42.594 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:42.595 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:42.595 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:42.595 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:42.595 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:42.596 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:42.596 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:42.596 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:42.596 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:42.596 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:42.597 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:42.597 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:42.597 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:42.597 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:42.597 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:42.598 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:42.598 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:42.598 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:42.599 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:42.599 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:42.599 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:42.599 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:42.600 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:42.600 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:42.601 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.470944_133.1 2023-10-12T19:36:47.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:47.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:47.604 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:47.683 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:47.683 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:47.855 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:47.856 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:47.856 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:47.856 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:47.856 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:47.857 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:47.857 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:47.857 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:47.857 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:47.857 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:47.857 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:47.858 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:47.858 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:47.858 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T19:36:47.858 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:47.859 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:47.859 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:47.859 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T19:36:47.859 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:47.859 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:47.859 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:47.860 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:47.860 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:47.860 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:47.860 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:47.860 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:47.861 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:47.861 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:47.861 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:47.861 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:47.861 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:47.862 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:47.862 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:47.862 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:47.862 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:47.862 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:47.863 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:47.863 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:47.863 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:47.863 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:47.863 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:47.865 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:47.865 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:47.865 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:47.865 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:47.867 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:47.867 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:47.867 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:47.867 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:47.868 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:47.868 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:47.868 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:47.868 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:47.868 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:47.869 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:47.869 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:47.869 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:47.869 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:47.869 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:47.871 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.473047_134.1 2023-10-12T19:36:52.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:52.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:52.873 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:52.917 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:52.917 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:52.970 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:52.971 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:52.971 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:52.971 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:52.971 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:52.971 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:52.972 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:52.972 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:52.972 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:52.972 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:52.972 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:52.973 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:52.973 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:52.973 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:36:52.973 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:52.973 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:52.974 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:52.974 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:36:52.974 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:52.974 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:52.974 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:52.975 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:52.975 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:52.975 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:52.976 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:52.976 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:52.976 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:52.976 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:52.976 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:52.976 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:52.977 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:52.977 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:52.977 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:52.977 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:52.977 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:52.978 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:52.978 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:52.978 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:52.979 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:52.979 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:52.980 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:52.980 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:52.980 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:52.980 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:52.980 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:52.981 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:52.981 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:52.981 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:52.981 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:52.981 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:52.981 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:52.982 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:52.982 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:52.982 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:52.982 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:52.982 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:52.983 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:52.983 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:52.983 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:52.983 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:52.983 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:52.984 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:52.984 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:52.984 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:52.985 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.470944_133.1 2023-10-12T19:36:57.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:57.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:36:57.987 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:36:58.039 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:36:58.040 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:36:58.098 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:36:58.098 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:36:58.098 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:36:58.098 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:36:58.099 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:36:58.099 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:36:58.099 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:58.099 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:58.100 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:36:58.100 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:58.100 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:36:58.100 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:58.100 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:58.100 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:36:58.101 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:58.101 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:36:58.101 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:58.101 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:36:58.101 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:58.102 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:58.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:58.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:58.102 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:36:58.103 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:58.103 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:58.103 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:36:58.103 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:58.103 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:58.104 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:58.104 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:36:58.104 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:58.104 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:58.104 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:58.105 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:58.105 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:36:58.105 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:58.105 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:58.105 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:58.106 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:58.106 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:58.106 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:58.106 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:58.106 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:58.107 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:58.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:58.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:36:58.107 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:36:58.107 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:36:58.108 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:36:58.108 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:36:58.108 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:36:58.108 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:36:58.108 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:36:58.109 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:36:58.109 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:36:58.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:58.109 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:58.109 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:36:58.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:36:58.110 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:36:58.110 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:36:58.110 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:36:58.110 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:36:58.110 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:36:58.112 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.470944_133.1 2023-10-12T19:37:03.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:03.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:03.114 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:03.156 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:03.156 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:03.209 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.473047_134.1", 2023-10-12T19:37:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.473047Z", 2023-10-12T19:37:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:03.221 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:03.222 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.473047_134.1 2023-10-12T19:37:08.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:08.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:08.225 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:08.327 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:08.328 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:08.584 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:08.584 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:08.585 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:08.585 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:08.585 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:08.585 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:08.586 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:08.586 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:08.586 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:08.586 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:08.587 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:08.587 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:08.587 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:08.587 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:37:08.587 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:08.588 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:08.588 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:08.588 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:37:08.588 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:08.588 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:08.589 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:08.589 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:08.589 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:08.589 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:08.590 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:08.590 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:08.590 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:08.591 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:08.591 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:08.591 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:08.591 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:08.592 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:08.592 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:08.592 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:08.592 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:08.592 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:08.593 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:08.593 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:08.593 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:08.593 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:08.593 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:08.594 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:08.594 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:08.594 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:08.594 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:08.595 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:08.595 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:08.595 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:08.595 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:08.595 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:08.596 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:08.596 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:08.596 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:08.596 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:08.596 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:08.597 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:08.597 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:08.597 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:08.597 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:08.597 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:08.598 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:08.598 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:08.598 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:08.598 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:08.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:37:13.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:13.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:13.602 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:13.694 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:13.694 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:13.967 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:13.967 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:13.967 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:13.967 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:13.968 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:13.968 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:13.968 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:13.968 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:13.968 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:13.969 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:13.969 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:13.969 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:13.969 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:13.970 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:37:13.970 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:13.970 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:13.970 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:13.970 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:37:13.971 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:13.971 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:13.971 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:13.971 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:13.971 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:13.971 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:13.972 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:13.972 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:13.972 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:13.972 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:13.972 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:13.973 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:13.973 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:13.973 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:13.973 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:13.973 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:13.974 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:13.974 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:13.974 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:13.975 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:13.975 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:13.975 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:13.975 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:13.975 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:13.976 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:13.976 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:13.976 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:13.976 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:13.977 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:13.977 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:13.977 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:13.977 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:13.977 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:13.978 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:13.978 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:13.978 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:13.978 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:13.978 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:13.979 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:13.979 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:13.979 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:13.979 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:13.979 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:13.980 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:13.980 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:13.980 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:13.981 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.467824_131.1 2023-10-12T19:37:18.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:18.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:18.984 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:19.057 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:19.057 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:19.423 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:19.424 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:19.424 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:19.424 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:19.424 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:19.424 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:19.425 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:19.425 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:19.425 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:19.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:19.425 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:19.426 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:19.426 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:19.426 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.764432_172.1", 2023-10-12T19:37:19.426 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:19.426 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:19.427 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:19.427 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.764432Z", 2023-10-12T19:37:19.427 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:19.427 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:19.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:19.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:19.428 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:19.428 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:19.428 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:19.428 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:19.428 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:19.429 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:19.429 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:19.429 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:19.429 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:19.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:19.430 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:19.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:19.430 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:19.430 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:19.430 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:19.431 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:19.431 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:19.431 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:19.431 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:19.431 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:19.432 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:19.432 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:19.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:19.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:19.432 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:19.433 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:19.433 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:19.433 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:19.433 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:19.433 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:19.434 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:19.434 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:19.434 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:19.434 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:19.434 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:19.435 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:19.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:19.435 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:19.435 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:19.435 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:19.435 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:19.436 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:19.437 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.764432_172.1 2023-10-12T19:37:24.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:24.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:24.439 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:24.507 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:24.507 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:24.562 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:24.562 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:24.563 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:24.563 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:24.564 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:24.564 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:24.564 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:24.565 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:24.565 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:24.565 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:24.566 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:24.566 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:24.567 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:24.567 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T19:37:24.567 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:24.568 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:24.568 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:24.568 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T19:37:24.568 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:24.569 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:24.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:24.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:24.569 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:24.569 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:24.570 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:24.570 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:24.570 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:24.570 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:24.570 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:24.571 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:24.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:24.571 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:24.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:24.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:24.572 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:24.572 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:24.572 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:24.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:24.572 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:24.573 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:24.573 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:24.573 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:24.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:24.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:24.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:24.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:24.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:24.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:24.574 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:24.574 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:24.575 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:24.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:24.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:24.575 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:24.575 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:24.576 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:24.576 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:24.576 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:24.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:24.576 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:24.576 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:24.577 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:24.577 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:24.577 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:24.578 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.438188_130.1 2023-10-12T19:37:29.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:29.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:29.581 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:29.626 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:29.627 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:29.681 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:29.681 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:29.681 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:29.682 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:29.682 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:29.682 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:29.682 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:29.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:29.683 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:29.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:29.683 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:29.683 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:29.684 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:29.684 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:37:29.684 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:29.684 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:29.684 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:29.685 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:37:29.685 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:29.685 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:29.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:29.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:29.685 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:29.686 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:29.686 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:29.686 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:29.686 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:29.687 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:29.687 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:29.687 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:29.687 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:29.687 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:29.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:29.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:29.688 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:29.689 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:29.689 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:29.689 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:29.689 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:29.689 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:29.690 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:29.690 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:29.690 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:29.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:29.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:29.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:29.691 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:29.691 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:29.691 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:29.691 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:29.692 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:29.692 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:29.692 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:29.692 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:29.692 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:29.692 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:29.693 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:29.693 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:29.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:29.693 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:29.693 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:29.694 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:29.694 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:29.694 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:29.695 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.470944_133.1 2023-10-12T19:37:34.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:34.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:34.698 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:34.742 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:34.742 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:34.794 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:34.794 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:34.794 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:34.795 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:34.795 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:34.795 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:34.795 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:34.795 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:34.796 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:34.796 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:34.796 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:34.796 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:34.796 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:34.796 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:37:34.797 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:34.797 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:34.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:34.797 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:37:34.797 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:34.798 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:34.798 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:34.798 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:34.799 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:34.799 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:34.799 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:34.799 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:34.799 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:34.800 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:34.800 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:34.800 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:34.800 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:34.800 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:34.801 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:34.801 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:34.802 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:34.802 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:34.802 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:34.802 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:34.802 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:34.803 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:34.803 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:34.803 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:34.803 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:34.803 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:34.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:34.804 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:34.804 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:34.804 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:34.804 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:34.805 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:34.805 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:34.805 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:34.805 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:34.805 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:34.805 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:34.806 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:34.806 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:34.806 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:34.806 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:34.806 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:34.807 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:34.807 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:34.807 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:34.807 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:34.808 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:37:39.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:39.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:39.811 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:39.853 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:39.854 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:39.908 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:39.909 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:39.909 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:39.909 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:39.909 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:39.909 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:39.910 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:39.910 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:39.910 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:39.910 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:39.910 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:39.911 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:39.911 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:39.911 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.438188_130.1", 2023-10-12T19:37:39.911 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:39.911 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.438188Z", 2023-10-12T19:37:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:39.921 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:39.922 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.438188_130.1 2023-10-12T19:37:44.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:44.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:44.925 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:44.968 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:44.969 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:45.020 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:45.021 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:45.021 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:45.021 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:45.021 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:45.022 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:45.022 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:45.022 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:45.022 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:45.023 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:45.023 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:45.023 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:45.023 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:45.023 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:37:45.024 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:45.024 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:45.024 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:45.024 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:37:45.025 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:45.025 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:45.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:45.025 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:45.025 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:45.026 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:45.026 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:45.026 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:45.026 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:45.029 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:45.029 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:45.030 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:45.030 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:45.030 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:45.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:45.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:45.031 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:45.031 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:45.031 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:45.031 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:45.032 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:45.032 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:45.032 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:45.032 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:45.033 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:45.033 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:45.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:45.033 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:45.034 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:45.034 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:45.034 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:45.034 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:45.034 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:45.035 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:45.035 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:45.035 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:45.035 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:45.035 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:45.036 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:45.036 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:45.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:45.036 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:45.037 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:45.037 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:45.037 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:45.037 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:45.038 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.436071_129.1 2023-10-12T19:37:50.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:50.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:50.040 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:50.096 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:50.096 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:50.148 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:50.148 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:50.149 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:50.149 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:50.149 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:50.149 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:50.149 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:50.150 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:50.150 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:50.150 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:50.150 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:50.150 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:50.151 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:50.151 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:37:50.151 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:50.151 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:50.151 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:50.152 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:37:50.152 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:50.152 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:50.152 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:50.152 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:50.153 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:50.153 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:50.153 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:50.153 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:50.153 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:50.154 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:50.154 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:50.154 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:50.154 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:50.154 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:50.155 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:50.155 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:50.155 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:50.156 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:50.156 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:50.156 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:50.156 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:50.156 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:50.157 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:50.157 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:50.157 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:50.157 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:50.157 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:50.158 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:50.158 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:50.158 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:50.158 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:50.158 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:50.159 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:50.159 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:50.159 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:50.159 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:50.159 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:50.160 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:50.160 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:50.160 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:50.160 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:50.160 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:50.161 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:50.161 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:50.161 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:50.161 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:50.162 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.470944_133.1 2023-10-12T19:37:55.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:55.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:37:55.164 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:37:55.251 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:37:55.252 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:37:55.567 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:37:55.568 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:37:55.568 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:37:55.568 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:37:55.568 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:37:55.568 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:37:55.569 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:55.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:55.569 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:37:55.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:55.569 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:37:55.569 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:55.570 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:55.570 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.435183_128.1", 2023-10-12T19:37:55.570 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:55.570 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:37:55.571 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:55.571 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.435183Z", 2023-10-12T19:37:55.571 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:55.571 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:55.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:55.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:55.572 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:37:55.572 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:55.572 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:55.572 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:37:55.572 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:55.573 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:55.573 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:55.573 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:37:55.573 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:55.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:55.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:55.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:55.574 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:37:55.574 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:55.574 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:55.574 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:55.575 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:55.575 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:55.575 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:55.575 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:55.575 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:55.575 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:55.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:55.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:37:55.576 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:37:55.576 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:37:55.576 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:37:55.577 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:37:55.577 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:37:55.577 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:37:55.577 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:37:55.577 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:37:55.577 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:37:55.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:55.578 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:55.578 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:37:55.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:37:55.578 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:37:55.579 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:37:55.579 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:37:55.579 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:37:55.579 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:37:55.581 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.435183_128.1 2023-10-12T19:38:00.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:00.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:00.584 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:00.631 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:00.631 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:00.733 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:00.734 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:00.734 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:00.734 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:00.734 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:00.735 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:00.735 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:00.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:00.735 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:00.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:00.736 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:00.736 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:00.736 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:00.736 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:38:00.736 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:00.737 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:00.737 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:00.737 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:38:00.737 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:00.737 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:00.738 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:00.738 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:00.738 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:00.738 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:00.738 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:00.739 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:00.739 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:00.739 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:00.739 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:00.739 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:00.740 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:00.740 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:00.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:00.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:00.740 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:00.741 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:00.741 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:00.741 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:00.741 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:00.741 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:00.742 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:00.742 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:00.742 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:00.742 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:00.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:00.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:00.743 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:00.743 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:00.743 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:00.743 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:00.744 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:00.744 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:00.744 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:00.744 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:00.745 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:00.745 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:00.745 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:00.745 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:00.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:00.745 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:00.746 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:00.746 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:00.746 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:00.746 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:00.747 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:38:05.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:05.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:05.749 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:05.792 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:05.793 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:05.876 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:05.877 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:05.877 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:05.877 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:05.877 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:05.877 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:05.878 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:05.878 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:05.878 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:05.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:05.878 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:05.879 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:05.879 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:05.879 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.610069_152.1", 2023-10-12T19:38:05.879 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:05.879 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:05.880 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:05.880 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.610069Z", 2023-10-12T19:38:05.880 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:05.880 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:05.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:05.881 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:05.881 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:05.881 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:05.881 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:05.881 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:05.881 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:05.882 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:05.882 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:05.882 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:05.882 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:05.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:05.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:05.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:05.883 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:05.883 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:05.884 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:05.884 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:05.884 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:05.884 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:05.884 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:05.884 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:05.885 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:05.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:05.885 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:05.885 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:05.885 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:05.886 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:05.886 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:05.886 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:05.886 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:05.886 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:05.887 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:05.887 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:05.887 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:05.887 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:05.887 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:05.888 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:05.888 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:05.888 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:05.888 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:05.888 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:05.889 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:05.889 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:05.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.610069_152.1 2023-10-12T19:38:10.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:10.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:10.893 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:10.975 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:10.976 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:11.240 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:38:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:38:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:11.253 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:11.254 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:38:16.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:16.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:16.257 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:16.300 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:16.300 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:16.602 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:16.602 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:16.602 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:16.602 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:16.603 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:16.603 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:16.603 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:16.603 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:16.603 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:16.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:16.604 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:16.604 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:16.604 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:16.604 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.436071_129.1", 2023-10-12T19:38:16.605 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:16.605 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:16.605 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:16.605 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.436071Z", 2023-10-12T19:38:16.605 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:16.606 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:16.606 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:16.606 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:16.606 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:16.606 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:16.607 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:16.607 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:16.607 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:16.607 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:16.608 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:16.608 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:16.608 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:16.608 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:16.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:16.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:16.609 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:16.609 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:16.609 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:16.609 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:16.610 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:16.610 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:16.610 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:16.610 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:16.610 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:16.611 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:16.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:16.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:16.611 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:16.611 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:16.612 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:16.612 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:16.612 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:16.612 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:16.612 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:16.613 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:16.613 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:16.613 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:16.613 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:16.613 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:16.613 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:16.614 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:16.614 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:16.614 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:16.614 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:16.614 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:16.615 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.436071_129.1 2023-10-12T19:38:21.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:21.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:21.618 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:21.660 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:21.660 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:21.713 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:21.713 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:21.713 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:21.713 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:21.714 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:21.714 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:21.714 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:21.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:21.714 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:21.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:21.715 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:21.715 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:21.715 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:21.715 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:38:21.716 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:21.716 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:21.716 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:21.716 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:38:21.716 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:21.717 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:21.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:21.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:21.717 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:21.717 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:21.718 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:21.718 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:21.718 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:21.718 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:21.718 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:21.719 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:21.719 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:21.719 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:21.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:21.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:21.720 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:21.720 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:21.720 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:21.720 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:21.720 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:21.721 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:21.721 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:21.721 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:21.721 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:21.721 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:21.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:21.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:21.722 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:21.722 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:21.722 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:21.723 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:21.723 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:21.723 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:21.723 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:21.723 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:21.724 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:21.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:21.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:21.724 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:21.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:21.725 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:21.725 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:21.725 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:21.725 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:21.725 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:21.727 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:38:26.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:26.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:26.730 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:26.779 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:26.779 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:26.836 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:26.836 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:26.836 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:26.836 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:26.837 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:26.837 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:26.837 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:26.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:26.837 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:26.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:26.838 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:26.838 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:26.838 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:26.838 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.432502_127.1", 2023-10-12T19:38:26.839 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:26.839 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:26.839 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:26.839 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.432502Z", 2023-10-12T19:38:26.839 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:26.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:26.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:26.840 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:26.840 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:26.840 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:26.841 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:26.841 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:26.841 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:26.841 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:26.841 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:26.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:26.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:26.845 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:26.845 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:26.845 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:26.845 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:26.846 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:26.847 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:26.848 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:26.848 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:26.848 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:26.849 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.432502_127.1 2023-10-12T19:38:31.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:31.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:31.852 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:31.894 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:31.895 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:31.948 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:31.948 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:31.948 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:31.949 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:31.949 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:31.949 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:31.949 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:31.949 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:31.950 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:31.950 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:31.950 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:31.950 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:31.950 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:31.951 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:38:31.951 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:31.951 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:31.951 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:31.951 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:38:31.952 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:31.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:31.952 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:31.952 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:31.952 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:31.953 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:31.953 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:31.953 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:31.953 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:31.953 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:31.953 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:31.954 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:31.954 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:31.954 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:31.954 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:31.954 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:31.955 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:31.955 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:31.955 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:31.955 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:31.955 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:31.956 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:31.956 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:31.956 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:31.956 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:31.956 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:31.957 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:31.957 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:31.957 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:31.957 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:31.957 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:31.957 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:31.958 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:31.958 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:31.958 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:31.958 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:31.959 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:31.959 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:31.959 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:31.959 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:31.959 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:31.959 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:31.960 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:31.960 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:31.960 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:31.960 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:31.961 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.407338_124.1 2023-10-12T19:38:36.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:36.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:36.963 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:37.007 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:37.008 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:37.062 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:37.063 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:37.063 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:37.063 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:37.064 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:37.064 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:37.064 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:37.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:37.065 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:37.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:37.065 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:37.065 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:37.066 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:37.066 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:38:37.066 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:37.066 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:37.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:37.067 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:38:37.067 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:37.067 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:37.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:37.068 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:37.068 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:37.068 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:37.068 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:37.068 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:37.069 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:37.069 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:37.069 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:37.069 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:37.070 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:37.070 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:37.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:37.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:37.070 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:37.071 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:37.071 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:37.071 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:37.071 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:37.071 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:37.072 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:37.072 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:37.072 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:37.073 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:37.073 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:37.073 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:37.073 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:37.074 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:37.074 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:37.074 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:37.074 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:37.075 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:37.075 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:37.075 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:37.075 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:37.075 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:37.076 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:37.076 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:37.076 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:37.076 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:37.076 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:37.077 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:37.077 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:37.077 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:37.077 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.470944_133.1 2023-10-12T19:38:42.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:42.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:42.080 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:42.148 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:42.148 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:42.201 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:42.201 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:42.201 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:42.201 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:42.202 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:42.202 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:42.202 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:42.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:42.202 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:42.203 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.611004_153.1", 2023-10-12T19:38:42.204 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:42.204 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:42.204 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:42.204 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.611004Z", 2023-10-12T19:38:42.204 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:42.205 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:42.206 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:42.207 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:42.207 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:42.207 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:42.207 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:42.208 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:42.211 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:42.212 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:42.212 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:42.212 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:42.212 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:42.212 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:42.213 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:42.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:42.213 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:42.213 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:42.213 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:42.213 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:42.214 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:42.215 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.611004_153.1 2023-10-12T19:38:47.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:47.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:47.218 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:47.298 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:47.299 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:47.602 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:47.603 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:47.603 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:47.603 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:47.604 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:47.604 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:47.604 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:47.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:47.604 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:47.605 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:47.605 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:47.605 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:47.605 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:47.605 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.613312_154.1", 2023-10-12T19:38:47.606 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:47.606 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:47.606 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:47.606 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.613312Z", 2023-10-12T19:38:47.606 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:47.607 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:47.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:47.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:47.607 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:47.607 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:47.608 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:47.608 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:47.608 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:47.608 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:47.608 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:47.609 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:47.609 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:47.609 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:47.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:47.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:47.610 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:47.610 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:47.610 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:47.610 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:47.610 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:47.611 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:47.611 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:47.611 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:47.611 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:47.611 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:47.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:47.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:47.612 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:47.612 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:47.612 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:47.613 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:47.613 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:47.613 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:47.613 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:47.613 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:47.614 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:47.614 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:47.614 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:47.614 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:47.614 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:47.615 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:47.615 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:47.615 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:47.615 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:47.615 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:47.616 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.613312_154.1 2023-10-12T19:38:52.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:52.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:52.619 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:52.664 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:52.664 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:52.725 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:52.726 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:52.726 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:52.726 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:52.726 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:52.727 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:52.727 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:52.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:52.727 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:52.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:52.728 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:52.728 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:52.728 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:52.728 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:38:52.728 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:52.729 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:52.729 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:52.729 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:38:52.729 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:52.729 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:52.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:52.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:52.730 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:52.730 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:52.730 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:52.731 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:52.731 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:52.731 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:52.731 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:52.731 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:52.732 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:52.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:52.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:52.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:52.732 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:52.733 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:52.733 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:52.733 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:52.733 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:52.733 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:52.734 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:52.734 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:52.734 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:52.734 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:52.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:52.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:52.735 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:52.735 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:52.735 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:52.735 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:52.736 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:52.736 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:52.736 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:52.736 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:52.736 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:52.737 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:52.737 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:52.737 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:52.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:52.737 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:52.737 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:52.738 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:52.738 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:52.738 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:52.739 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.467824_131.1 2023-10-12T19:38:57.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:57.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:38:57.741 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:38:57.842 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:38:57.843 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:38:58.073 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:38:58.074 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:38:58.074 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:38:58.074 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:38:58.074 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:38:58.074 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:38:58.075 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:58.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:58.075 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:38:58.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:58.075 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:38:58.076 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:58.076 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:58.076 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:38:58.076 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:58.076 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:38:58.077 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:58.077 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:38:58.077 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:58.077 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:58.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:58.078 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:58.078 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:38:58.078 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:58.078 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:58.078 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:38:58.079 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:58.079 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:58.079 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:58.079 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:38:58.079 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:58.080 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:58.080 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:58.080 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:58.080 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:38:58.080 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:58.081 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:58.081 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:58.081 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:58.081 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:58.081 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:58.082 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:58.082 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:58.082 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:58.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:58.082 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:38:58.082 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:38:58.083 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:38:58.083 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:38:58.083 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:38:58.083 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:38:58.083 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:38:58.084 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:38:58.084 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:38:58.084 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:38:58.084 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:58.084 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:58.088 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:38:58.088 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:38:58.089 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:38:58.089 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:38:58.089 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:38:58.089 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:38:58.089 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:38:58.091 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:39:03.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:03.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:03.093 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:03.141 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:03.142 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:03.571 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:03.572 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:03.572 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:03.572 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:03.572 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:03.572 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:03.573 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:03.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:03.573 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:03.573 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:03.573 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:03.574 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:39:03.574 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:03.574 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:03.574 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:03.574 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:39:03.575 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:03.575 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:03.575 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:03.575 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:03.575 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:03.576 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:03.576 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:03.576 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:03.576 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:03.576 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:03.577 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:03.577 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:03.577 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:03.577 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:03.577 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:03.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:03.578 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:03.578 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:03.578 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:03.578 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:03.579 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:03.579 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:03.579 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:03.579 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:03.579 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:03.580 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:03.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:03.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:03.580 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:03.581 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:03.581 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:03.581 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:03.581 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:03.581 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:03.582 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:03.582 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:03.582 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:03.582 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:03.582 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:03.583 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:03.583 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:03.583 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:03.583 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:03.583 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:03.584 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:03.584 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:03.584 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:39:08.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:08.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:08.586 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:08.684 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:08.684 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:08.684 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:08.685 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:08.685 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:08.685 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:08.685 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:08.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:08.686 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:08.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:08.686 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:08.686 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:08.687 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:08.687 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.470944_133.1", 2023-10-12T19:39:08.687 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:08.687 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:08.687 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:08.687 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.470944Z", 2023-10-12T19:39:08.688 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:08.688 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:08.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:08.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:08.688 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:08.689 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:08.689 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:08.689 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:08.689 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:08.689 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:08.690 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:08.690 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:08.690 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:08.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:08.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:08.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:08.691 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:08.691 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:08.691 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:08.691 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:08.692 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:08.692 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:08.692 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:08.692 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:08.692 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:08.693 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:08.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:08.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:08.693 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:08.693 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:08.693 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:08.694 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:08.694 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:08.694 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:08.694 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:08.694 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:08.695 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:08.695 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:08.695 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:08.695 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:08.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:08.696 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:08.696 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:08.696 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:08.696 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:08.696 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:08.697 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.470944_133.1 2023-10-12T19:39:13.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:13.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:13.700 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:13.742 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:13.742 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:13.795 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:13.796 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:13.796 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:13.796 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:13.797 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:13.797 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:13.797 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:13.797 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:13.797 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:13.797 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:13.798 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:13.798 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:13.798 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:13.798 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.568332_143.1", 2023-10-12T19:39:13.798 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:13.799 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:13.799 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:13.799 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.568332Z", 2023-10-12T19:39:13.799 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:13.799 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:13.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:13.800 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:13.800 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:13.800 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:13.800 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:13.800 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:13.801 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:13.801 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:13.801 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:13.801 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:13.801 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:13.802 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:13.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:13.802 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:13.802 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:13.803 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:13.803 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:13.803 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:13.803 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:13.803 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:13.803 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:13.804 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:13.804 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:13.804 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:13.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:13.804 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:13.805 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:13.805 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:13.805 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:13.805 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:13.805 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:13.806 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:13.806 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:13.806 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:13.806 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:13.806 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:13.807 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:13.807 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:13.807 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:13.807 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:13.807 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:13.808 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:13.808 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:13.808 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:13.809 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.568332_143.1 2023-10-12T19:39:18.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:18.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:18.812 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:18.900 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:18.901 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:19.158 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:19.159 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:19.159 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:19.159 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:19.159 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:19.160 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:19.160 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:19.160 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:19.160 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:19.160 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:39:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:39:19.162 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:19.163 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:19.164 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:19.165 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:19.165 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:19.165 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:19.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:19.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:19.166 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:19.166 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:19.166 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:19.166 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:19.167 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:19.167 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:19.167 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:19.167 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:19.167 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:19.168 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:19.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:19.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:19.168 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:19.168 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:19.169 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:19.169 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:19.169 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:19.169 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:19.169 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:19.170 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:19.170 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:19.170 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:19.170 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:19.170 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:19.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:19.171 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:19.171 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:19.171 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:19.171 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:19.172 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:19.172 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:39:24.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:24.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:24.174 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:24.262 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:24.263 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:24.440 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:24.441 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:24.441 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:24.441 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:24.442 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:24.442 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:24.442 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:24.442 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:24.442 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:24.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:24.443 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:24.443 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:24.443 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:24.443 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:39:24.444 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:24.444 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:24.444 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:24.444 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:39:24.444 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:24.445 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:24.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:24.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:24.445 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:24.445 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:24.445 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:24.446 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:24.446 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:24.446 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:24.446 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:24.446 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:24.447 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:24.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:24.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:24.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:24.447 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:24.448 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:24.448 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:24.448 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:24.448 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:24.448 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:24.449 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:24.449 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:24.449 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:24.449 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:24.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:24.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:24.450 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:24.450 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:24.450 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:24.451 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:24.451 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:24.451 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:24.451 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:24.451 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:24.451 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:24.452 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:24.452 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:24.452 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:24.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:24.452 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:24.453 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:24.453 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:24.453 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:24.453 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:24.454 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:39:29.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:29.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:29.457 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:29.538 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:29.539 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:29.828 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:29.828 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:29.828 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:29.828 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:29.828 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:29.829 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:29.829 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:29.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:29.829 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:29.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:29.830 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:29.830 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:29.830 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:29.830 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.407338_124.1", 2023-10-12T19:39:29.830 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.407338Z", 2023-10-12T19:39:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:29.831 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:29.832 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:29.833 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:29.834 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:29.835 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:29.836 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:29.837 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:29.838 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:29.839 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:29.840 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:29.841 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.407338_124.1 2023-10-12T19:39:34.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:34.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:34.844 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:34.920 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:34.920 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:34.975 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:34.976 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:34.976 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:34.976 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:34.976 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:34.977 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:34.977 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:34.977 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:34.977 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:34.977 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:34.978 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:34.978 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:34.978 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:34.978 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:39:34.978 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:34.979 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:34.979 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:34.979 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:39:34.979 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:34.979 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:34.980 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:34.980 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:34.980 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:34.980 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:34.980 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:34.980 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:34.981 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:34.981 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:34.981 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:34.981 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:34.981 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:34.982 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:34.982 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:34.982 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:34.982 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:34.982 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:34.983 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:34.983 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:34.983 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:34.983 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:34.983 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:34.983 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:34.984 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:34.984 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:34.984 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:34.984 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:34.984 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:34.985 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:34.985 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:34.985 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:34.985 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:34.985 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:34.985 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:34.986 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:34.986 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:34.986 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:34.986 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:34.986 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:34.987 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:34.987 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:34.987 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:34.987 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:34.987 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:34.988 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:34.989 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:39:39.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:39.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:39.991 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:40.082 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:40.083 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:40.258 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:40.258 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:40.259 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:40.259 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:40.259 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:40.259 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:40.261 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:40.261 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:40.261 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.547705_140.1", 2023-10-12T19:39:40.261 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:40.261 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.547705Z", 2023-10-12T19:39:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:40.264 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:40.264 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:40.264 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:40.264 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:40.264 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:40.267 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:40.267 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:40.267 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:40.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:40.267 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:40.271 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:40.271 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:40.272 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.547705_140.1 2023-10-12T19:39:45.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:45.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:45.274 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:45.318 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:45.318 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:45.374 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:45.374 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:45.374 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:45.374 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:45.375 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:45.375 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:45.375 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:45.375 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:45.375 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:45.376 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:45.376 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:45.376 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:45.376 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:45.377 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.410419_126.1", 2023-10-12T19:39:45.377 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:45.377 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:45.377 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:45.377 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.410419Z", 2023-10-12T19:39:45.378 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:45.378 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:45.378 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:45.378 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:45.378 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:45.379 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:45.379 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:45.379 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:45.379 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:45.380 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:45.380 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:45.380 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:45.380 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:45.380 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:45.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:45.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:45.381 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:45.381 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:45.381 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:45.382 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:45.382 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:45.382 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:45.382 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:45.382 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:45.383 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:45.383 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:45.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:45.383 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:45.384 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:45.384 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:45.384 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:45.384 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:45.384 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:45.385 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:45.385 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:45.385 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:45.385 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:45.385 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:45.386 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:45.386 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:45.386 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:45.386 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:45.386 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:45.387 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:45.387 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:45.387 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:45.388 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.410419_126.1 2023-10-12T19:39:50.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:50.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:50.391 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:50.434 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:50.435 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:50.487 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:50.487 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:50.488 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:50.488 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:50.488 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:50.488 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:50.488 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:50.489 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:50.489 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:50.489 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:50.489 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:50.489 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:50.490 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:50.490 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:39:50.490 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:50.490 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:50.490 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:50.491 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:39:50.491 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:50.491 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:50.491 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:50.491 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:50.492 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:50.492 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:50.492 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:50.492 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:50.492 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:50.493 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:50.493 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:50.493 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:50.493 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:50.493 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:50.493 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:50.494 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:50.494 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:50.494 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:50.494 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:50.494 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:50.495 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:50.495 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:50.495 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:50.495 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:50.495 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:50.496 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:50.496 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:50.496 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:50.496 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:50.496 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:50.497 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:50.497 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:50.497 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:50.497 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:50.497 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:50.497 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:50.498 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:50.498 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:50.498 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:50.498 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:50.498 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:50.499 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:50.499 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:50.499 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:50.499 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:50.499 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:50.501 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.467824_131.1 2023-10-12T19:39:55.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:55.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:39:55.503 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:39:55.546 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:39:55.546 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:39:55.600 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:39:55.600 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:39:55.600 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:39:55.601 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:39:55.601 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:39:55.601 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:39:55.601 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:55.601 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:55.602 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:39:55.602 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:55.602 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:39:55.602 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:55.603 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:55.603 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:39:55.603 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:55.603 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:39:55.603 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:55.604 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:39:55.604 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:55.604 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:55.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:55.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:55.605 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:39:55.605 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:55.605 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:55.605 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:39:55.605 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:55.606 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:55.606 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:55.606 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:39:55.606 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:55.607 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:55.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:55.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:55.608 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:39:55.608 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:55.608 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:55.609 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:55.609 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:55.609 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:55.609 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:55.609 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:55.610 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:55.610 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:55.610 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:55.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:39:55.611 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:39:55.611 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:39:55.611 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:39:55.611 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:39:55.612 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:39:55.612 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:39:55.612 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:39:55.612 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:39:55.612 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:39:55.613 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:55.613 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:55.613 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:39:55.613 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:39:55.614 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:39:55.614 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:39:55.614 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:39:55.615 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:39:55.615 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:39:55.616 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:40:00.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:00.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:00.619 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:40:00.664 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:40:00.665 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:40:00.715 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:40:00.715 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:40:00.716 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:40:00.716 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:40:00.716 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:40:00.716 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:40:00.716 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:00.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:00.717 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:40:00.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:00.717 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:40:00.717 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:00.717 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:00.718 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:40:00.718 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:00.718 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:40:00.718 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:00.719 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:40:00.719 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:00.719 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:00.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:00.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:00.720 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:40:00.720 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:00.721 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:00.721 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:40:00.721 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:00.721 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:00.721 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:00.722 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:40:00.722 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:00.722 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:00.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:00.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:00.723 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:40:00.723 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:00.723 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:00.723 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:00.723 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:00.724 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:00.724 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:00.724 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:00.724 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:00.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:00.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:00.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:00.725 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:40:00.725 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:00.725 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:00.726 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:00.726 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:00.726 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:00.726 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:00.726 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:00.727 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:00.727 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:00.727 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:00.727 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:40:00.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:00.728 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:40:00.728 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:40:00.728 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:40:00.728 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:00.728 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:40:00.729 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:40:05.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:05.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:05.732 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:40:05.774 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:40:05.774 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:40:05.862 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:40:05.862 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:40:05.862 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:40:05.863 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:40:05.863 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:40:05.863 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:40:05.863 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:05.863 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:05.864 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:40:05.864 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:05.864 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:40:05.864 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:05.864 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:05.864 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.486122_135.1", 2023-10-12T19:40:05.865 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:05.865 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:40:05.865 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:05.865 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.486122Z", 2023-10-12T19:40:05.865 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:05.866 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:05.866 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:05.866 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:05.866 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:40:05.866 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:05.867 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:05.867 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:40:05.867 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:05.867 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:05.867 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:05.868 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:40:05.868 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:05.868 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:05.868 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:05.868 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:05.869 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:40:05.869 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:05.869 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:05.869 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:05.869 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:05.870 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:05.870 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:05.870 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:05.870 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:05.871 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:05.871 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:05.871 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:05.871 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:40:05.871 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:05.872 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:05.872 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:05.872 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:05.872 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:05.872 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:05.873 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:05.873 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:05.873 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:05.873 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:05.873 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:40:05.874 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:05.874 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:40:05.874 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:40:05.874 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:40:05.875 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:05.875 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:40:05.876 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.486122_135.1 2023-10-12T19:40:10.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:10.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:10.878 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:40:10.967 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:40:10.967 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:40:11.079 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:40:11.080 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:40:11.080 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:40:11.080 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:40:11.080 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:40:11.081 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:40:11.081 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:11.081 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:11.081 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:40:11.081 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:11.082 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:40:11.082 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:11.082 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:11.082 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:40:11.082 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:11.083 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:40:11.083 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:11.083 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:40:11.083 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:11.083 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:11.083 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:11.084 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:11.084 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:40:11.084 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:11.084 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:11.084 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:40:11.085 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:11.085 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:11.085 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:11.085 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:40:11.085 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:11.086 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:11.086 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:11.086 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:11.086 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:40:11.086 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:11.087 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:11.087 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:11.087 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:11.087 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:11.087 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:11.088 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:11.088 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:11.089 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:11.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:11.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:11.090 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:40:11.090 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:11.090 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:11.091 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:11.091 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:11.091 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:11.092 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:11.092 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:11.092 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:11.093 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:11.093 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:11.094 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:40:11.094 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:11.094 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:40:11.095 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:40:11.095 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:40:11.095 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:11.096 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:40:11.098 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:40:16.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:16.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:16.100 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:40:16.148 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:40:16.148 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:40:16.208 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:40:16.208 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:40:16.208 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:40:16.209 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:40:16.209 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:40:16.209 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:40:16.209 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:16.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:16.210 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:40:16.210 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:16.210 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:40:16.210 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:16.210 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:40:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:40:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:16.211 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:40:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:40:16.212 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:40:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:16.213 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:16.214 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:40:16.214 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:16.214 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:16.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:16.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:40:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:16.215 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:16.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:16.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:16.217 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:40:16.217 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:16.217 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:16.217 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:16.218 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:40:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:40:16.219 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:40:16.220 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:40:16.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:16.220 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:40:16.221 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.467824_131.1 2023-10-12T19:40:21.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:21.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:21.223 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:40:21.292 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:40:21.292 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:40:21.349 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:40:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:40:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:40:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:40:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:40:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:40:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:40:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:40:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.467824_131.1", 2023-10-12T19:40:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:40:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.467824Z", 2023-10-12T19:40:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:40:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:40:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:40:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:40:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:40:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:40:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:40:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:40:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:40:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:21.362 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:40:21.363 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.467824_131.1 2023-10-12T19:40:26.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:26.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T19:40:26.366 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T19:40:26.469 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setuser ceph since I am not root 2023-10-12T19:40:26.469 INFO:teuthology.orchestra.run.smithi170.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T19:40:26.737 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-12T19:40:26.738 INFO:teuthology.orchestra.run.smithi170.stdout: "sync_status": { 2023-10-12T19:40:26.738 INFO:teuthology.orchestra.run.smithi170.stdout: "info": { 2023-10-12T19:40:26.738 INFO:teuthology.orchestra.run.smithi170.stdout: "status": "sync", 2023-10-12T19:40:26.738 INFO:teuthology.orchestra.run.smithi170.stdout: "num_shards": 4, 2023-10-12T19:40:26.739 INFO:teuthology.orchestra.run.smithi170.stdout: "period": "4fe09484-838f-4904-82b9-13168a9d62ca", 2023-10-12T19:40:26.739 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:26.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:26.739 INFO:teuthology.orchestra.run.smithi170.stdout: "markers": [ 2023-10-12T19:40:26.739 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:26.740 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 0, 2023-10-12T19:40:26.740 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:26.740 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:26.740 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136751.408247_125.1", 2023-10-12T19:40:26.740 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:26.741 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 1, 2023-10-12T19:40:26.741 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:26.741 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:52:31.408247Z", 2023-10-12T19:40:26.741 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:26.741 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:26.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:26.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:26.742 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 1, 2023-10-12T19:40:26.742 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:26.742 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:26.743 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "1_1697136633.545339_103.1", 2023-10-12T19:40:26.743 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:26.743 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:26.743 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:26.744 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "2023-10-12T18:50:33.545339Z", 2023-10-12T19:40:26.744 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:26.744 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:26.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:26.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:26.745 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 2, 2023-10-12T19:40:26.745 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:26.745 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:26.745 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:26.745 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:26.746 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:26.746 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:26.746 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:26.746 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:26.746 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:26.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:26.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-12T19:40:26.747 INFO:teuthology.orchestra.run.smithi170.stdout: "key": 3, 2023-10-12T19:40:26.747 INFO:teuthology.orchestra.run.smithi170.stdout: "val": { 2023-10-12T19:40:26.747 INFO:teuthology.orchestra.run.smithi170.stdout: "state": 1, 2023-10-12T19:40:26.748 INFO:teuthology.orchestra.run.smithi170.stdout: "marker": "", 2023-10-12T19:40:26.748 INFO:teuthology.orchestra.run.smithi170.stdout: "next_step_marker": "", 2023-10-12T19:40:26.748 INFO:teuthology.orchestra.run.smithi170.stdout: "total_entries": 0, 2023-10-12T19:40:26.748 INFO:teuthology.orchestra.run.smithi170.stdout: "pos": 0, 2023-10-12T19:40:26.748 INFO:teuthology.orchestra.run.smithi170.stdout: "timestamp": "0.000000", 2023-10-12T19:40:26.748 INFO:teuthology.orchestra.run.smithi170.stdout: "realm_epoch": 2 2023-10-12T19:40:26.749 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:26.749 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:26.749 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-12T19:40:26.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-12T19:40:26.749 INFO:teuthology.orchestra.run.smithi170.stdout: "full_sync": { 2023-10-12T19:40:26.750 INFO:teuthology.orchestra.run.smithi170.stdout: "total": 1, 2023-10-12T19:40:26.750 INFO:teuthology.orchestra.run.smithi170.stdout: "complete": 1 2023-10-12T19:40:26.750 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-12T19:40:26.750 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-12T19:40:26.751 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697139317.832704_330.1 target=1_1697136751.408247_125.1 2023-10-12T19:40:31.753 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2023-10-12T19:40:31.847 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... FAIL 2023-10-12T19:40:31.847 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.848 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.848 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_delete_notempty 2023-10-12T19:40:31.848 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.849 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.849 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.849 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.850 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 834, in test_bucket_delete_notempty 2023-10-12T19:40:31.850 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-10-12T19:40:31.850 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2023-10-12T19:40:31.850 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-10-12T19:40:31.851 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2023-10-12T19:40:31.851 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-10-12T19:40:31.851 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-10-12T19:40:31.852 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.852 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.852 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_datalog_autotrim 2023-10-12T19:40:31.853 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.853 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.853 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.854 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.854 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 952, in test_datalog_autotrim 2023-10-12T19:40:31.854 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-10-12T19:40:31.854 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2023-10-12T19:40:31.855 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-10-12T19:40:31.855 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2023-10-12T19:40:31.855 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-10-12T19:40:31.856 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-10-12T19:40:31.856 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.856 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.857 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_multi_zone_redirect 2023-10-12T19:40:31.857 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.857 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.857 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.858 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.858 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 995, in test_multi_zone_redirect 2023-10-12T19:40:31.858 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-10-12T19:40:31.859 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2023-10-12T19:40:31.859 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-10-12T19:40:31.859 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2023-10-12T19:40:31.859 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-10-12T19:40:31.860 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-10-12T19:40:31.860 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.860 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.861 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_zonegroup_remove 2023-10-12T19:40:31.861 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.861 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.862 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.862 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.862 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 1024, in test_zonegroup_remove 2023-10-12T19:40:31.862 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-10-12T19:40:31.863 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2023-10-12T19:40:31.863 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-10-12T19:40:31.863 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2023-10-12T19:40:31.865 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-10-12T19:40:31.865 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-10-12T19:40:31.866 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.866 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.866 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable 2023-10-12T19:40:31.866 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.867 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.867 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.867 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.868 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 1106, in test_bucket_sync_disable 2023-10-12T19:40:31.868 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-10-12T19:40:31.868 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2023-10-12T19:40:31.868 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-10-12T19:40:31.869 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2023-10-12T19:40:31.869 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-10-12T19:40:31.869 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-10-12T19:40:31.870 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.870 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.870 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2023-10-12T19:40:31.871 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.871 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.871 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.871 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.872 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 1129, in test_bucket_sync_enable_right_after_disable 2023-10-12T19:40:31.872 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-10-12T19:40:31.872 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2023-10-12T19:40:31.873 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-10-12T19:40:31.873 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2023-10-12T19:40:31.873 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-10-12T19:40:31.873 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-10-12T19:40:31.874 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.874 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.874 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable_enable 2023-10-12T19:40:31.875 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.875 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.875 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.876 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.876 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 1163, in test_bucket_sync_disable_enable 2023-10-12T19:40:31.876 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-10-12T19:40:31.876 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2023-10-12T19:40:31.877 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-10-12T19:40:31.877 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2023-10-12T19:40:31.877 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-10-12T19:40:31.878 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-10-12T19:40:31.878 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.878 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.878 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_multipart_object_sync 2023-10-12T19:40:31.879 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.879 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.879 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.880 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.880 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 1207, in test_multipart_object_sync 2023-10-12T19:40:31.880 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-12T19:40:31.880 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 387, in zonegroup_bucket_checkpoint 2023-10-12T19:40:31.881 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-12T19:40:31.881 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 379, in zone_bucket_checkpoint 2023-10-12T19:40:31.881 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-12T19:40:31.882 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-12T19:40:31.882 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-12T19:40:31.882 INFO:tasks.rgw_multisite_tests:AssertionError 2023-10-12T19:40:31.882 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.883 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.883 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_encrypted_object_sync 2023-10-12T19:40:31.883 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.884 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.884 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.884 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.885 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 1243, in test_encrypted_object_sync 2023-10-12T19:40:31.885 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-10-12T19:40:31.885 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2023-10-12T19:40:31.885 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-10-12T19:40:31.886 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2023-10-12T19:40:31.886 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-10-12T19:40:31.886 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-10-12T19:40:31.887 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.887 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.887 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_index_log_trim 2023-10-12T19:40:31.888 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.888 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.888 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.888 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.889 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 1273, in test_bucket_index_log_trim 2023-10-12T19:40:31.889 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2023-10-12T19:40:31.889 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 1268, in make_test_bucket 2023-10-12T19:40:31.889 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-10-12T19:40:31.890 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2023-10-12T19:40:31.890 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-10-12T19:40:31.890 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2023-10-12T19:40:31.891 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-10-12T19:40:31.891 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-10-12T19:40:31.891 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.891 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T19:40:31.892 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_creation_time 2023-10-12T19:40:31.892 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.892 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T19:40:31.893 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T19:40:31.893 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T19:40:31.893 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multi/tests.py", line 1315, in test_bucket_creation_time 2023-10-12T19:40:31.893 INFO:tasks.rgw_multisite_tests: eq(a.name, b.name) 2023-10-12T19:40:31.894 INFO:tasks.rgw_multisite_tests:AssertionError: 'pqrzfz-39' != 'pqrzfz-4' 2023-10-12T19:40:31.894 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.894 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T19:40:31.895 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 3032.425s 2023-10-12T19:40:31.895 INFO:tasks.rgw_multisite_tests: 2023-10-12T19:40:31.895 INFO:tasks.rgw_multisite_tests:FAILED (failures=11) 2023-10-12T19:40:31.896 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multisite_tests.py", line 71, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, test failures 2023-10-12T19:40:31.969 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0882e8ebe0604dd3a24e8ead69eab402 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_cbodley_ceph_e5d3c6c93855bbe531cbf7186fc3967731bde287/qa/tasks/rgw_multisite_tests.py", line 71, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, test failures 2023-10-12T19:40:31.971 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-10-12T19:40:31.981 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-10-12T19:40:31.997 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-12T19:40:32.012 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-12T19:40:32.012 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:40:32.017 INFO:tasks.rgw.c1.client.0.smithi170.stdout:2023-10-12T19:40:32.013+0000 7fb9a22ea700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 91079) UID: 0 2023-10-12T19:40:32.017 INFO:tasks.rgw.c1.client.0.smithi170.stdout:2023-10-12T19:40:32.013+0000 7fb9babac5c0 -1 shutting down 2023-10-12T19:40:35.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-12T19:40:38.114 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-12T19:40:38.115 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-10-12T19:40:38.153 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/vault-root-token 2023-10-12T19:40:38.309 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T19:40:38.309 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:40:38.310 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T19:40:38.308+0000 7f664de87700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 92283) UID: 0 2023-10-12T19:40:38.334 INFO:tasks.rgw.c1.client.1.smithi170.stdout:2023-10-12T19:40:38.308+0000 7f66667495c0 -1 shutting down 2023-10-12T19:40:40.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T19:40:40.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-12T19:40:44.411 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T19:40:44.412 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-10-12T19:40:44.428 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/vault-root-token 2023-10-12T19:40:44.493 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-12T19:40:44.494 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:40:44.495 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2023-10-12T19:40:44.494+0000 7f76153b5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 91063) UID: 0 2023-10-12T19:40:44.589 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2023-10-12T19:40:44.494+0000 7f762dc775c0 -1 shutting down 2023-10-12T19:40:46.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T19:40:47.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-12T19:40:50.595 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-12T19:40:50.596 DEBUG:teuthology.orchestra.run.smithi187:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-10-12T19:40:50.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/vault-root-token 2023-10-12T19:40:51.147 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-12T19:40:51.147 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:40:51.148 INFO:tasks.rgw.c2.client.1.smithi187.stdout:2023-10-12T19:40:51.147+0000 7fcef6a69700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 92259) UID: 0 2023-10-12T19:40:51.149 INFO:tasks.rgw.c2.client.1.smithi187.stdout:2023-10-12T19:40:51.147+0000 7fcf0f32b5c0 -1 shutting down 2023-10-12T19:40:52.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-12T19:40:52.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-12T19:40:57.249 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-12T19:40:57.250 DEBUG:teuthology.orchestra.run.smithi187:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-10-12T19:40:57.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/vault-root-token 2023-10-12T19:40:57.336 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-12T19:40:57.347 INFO:tasks.ceph.ceph_manager.c2:Canceling any pending splits or merges... 2023-10-12T19:40:57.347 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-12T19:40:57.759 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-12T19:40:57.760 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"fsid":"171d6df1-cef5-40ec-977b-bbc477f9da52","created":"2023-10-12T18:46:08.198710+0000","modified":"2023-10-12T18:50:23.216162+0000","last_up_change":"2023-10-12T18:46:15.493577+0000","last_in_change":"2023-10-12T18:46:09.118839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-12T18:46:14.529244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T18:46:16.798729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-10-12T18:46:30.432757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-10-12T18:46:32.517494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-10-12T18:46:45.576346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-10-12T18:46:47.277397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-10-12T18:46:49.256736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-10-12T18:46:51.309732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"b1.rgw.buckets.data","create_time":"2023-10-12T18:47:29.053593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"b1.rgw.log","create_time":"2023-10-12T18:47:30.481046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"b1.rgw.control","create_time":"2023-10-12T18:47:32.354299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"b1.rgw.meta","create_time":"2023-10-12T18:47:34.389626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"b2.rgw.buckets.data","create_time":"2023-10-12T18:47:45.109117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"b2.rgw.log","create_time":"2023-10-12T18:47:46.536330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"b2.rgw.control","create_time":"2023-10-12T18:47:48.397844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"b2.rgw.meta","create_time":"2023-10-12T18:47:50.444469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"b1.rgw.buckets.index","create_time":"2023-10-12T18:50:15.118285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"b2.rgw.buckets.index","create_time":"2023-10-12T18:50:20.559255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"94cdd9ee-c5f5-4302-8aa5-05e5993dd79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":86277},{"type":"v1","addr":"172.21.15.187:6801","nonce":86277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":86277},{"type":"v1","addr":"172.21.15.187:6803","nonce":86277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":86277},{"type":"v1","addr":"172.21.15.187:6807","nonce":86277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":86277},{"type":"v1","addr":"172.21.15.187:6805","nonce":86277}]},"public_addr":"172.21.15.187:6801/86277","cluster_addr":"172.21.15.187:6803/86277","heartbeat_back_addr":"172.21.15.187:6807/86277","heartbeat_front_addr":"172.21.15.187:6805/86277","state":["exists","up"]},{"osd":1,"uuid":"468de964-ef0f-4312-8eb7-8fe606aa4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":86292},{"type":"v1","addr":"172.21.15.187:6809","nonce":86292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":86292},{"type":"v1","addr":"172.21.15.187:6811","nonce":86292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":86292},{"type":"v1","addr":"172.21.15.187:6815","nonce":86292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":86292},{"type":"v1","addr":"172.21.15.187:6813","nonce":86292}]},"public_addr":"172.21.15.187:6809/86292","cluster_addr":"172.21.15.187:6811/86292","heartbeat_back_addr":"172.21.15.187:6815/86292","heartbeat_front_addr":"172.21.15.187:6813/86292","state":["exists","up"]},{"osd":2,"uuid":"184e7d87-7a36-412d-8bee-6e3391d5909c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":86293},{"type":"v1","addr":"172.21.15.187:6817","nonce":86293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":86293},{"type":"v1","addr":"172.21.15.187:6819","nonce":86293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":86293},{"type":"v1","addr":"172.21.15.187:6823","nonce":86293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":86293},{"type":"v1","addr":"172.21.15.187:6821","nonce":86293}]},"public_addr":"172.21.15.187:6817/86293","cluster_addr":"172.21.15.187:6819/86293","heartbeat_back_addr":"172.21.15.187:6823/86293","heartbeat_front_addr":"172.21.15.187:6821/86293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.465248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.456915+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:46:13.507348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1","mappings":[{"from":0,"to":1}]},{"pgid":"3.20","mappings":[{"from":0,"to":1}]},{"pgid":"4.d","mappings":[{"from":1,"to":0}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.26","mappings":[{"from":1,"to":0}]},{"pgid":"4.28","mappings":[{"from":1,"to":0}]},{"pgid":"4.2a","mappings":[{"from":1,"to":2}]},{"pgid":"4.37","mappings":[{"from":1,"to":2}]},{"pgid":"4.3b","mappings":[{"from":1,"to":2}]},{"pgid":"10.f","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T19:40:57.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-10-12T19:40:58.076 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-12T19:40:58.077 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-12T19:40:58.077 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-10-12T19:40:58.077 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:40:58.126 INFO:tasks.ceph.c2.osd.0:Stopped 2023-10-12T19:40:58.127 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-10-12T19:40:58.127 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:40:58.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-12T19:40:58.189 INFO:tasks.ceph.c2.osd.1:Stopped 2023-10-12T19:40:58.189 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-10-12T19:40:58.190 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:40:58.261 INFO:tasks.ceph.c2.osd.2:Stopped 2023-10-12T19:40:58.262 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-12T19:40:58.262 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-10-12T19:40:58.263 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:40:58.285 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-10-12T19:40:58.285 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-12T19:40:58.286 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-10-12T19:40:58.286 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:40:58.343 INFO:tasks.ceph.c2.mon.a:Stopped 2023-10-12T19:40:58.343 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-12T19:40:58.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-12T19:40:58.415 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T19:40:58.415 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-10-12T19:40:59.865 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T19:40:59.866 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-10-12T19:41:00.022 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T19:41:00.023 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-10-12T19:41:00.194 INFO:tasks.ceph:Archiving mon data... 2023-10-12T19:41:00.195 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584/data/c2.mon.a.tgz 2023-10-12T19:41:00.196 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2023-10-12T19:41:00.216 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.7B0AjqQyy9 2023-10-12T19:41:00.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.7B0AjqQyy9 2023-10-12T19:41:00.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.7B0AjqQyy9 2023-10-12T19:41:00.700 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.7B0AjqQyy9 is 1MB 2023-10-12T19:41:00.813 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.7B0AjqQyy9 2023-10-12T19:41:00.828 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-12T19:41:00.828 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-12T19:41:00.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-12T19:41:00.892 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-12T19:41:00.902 INFO:tasks.ceph.ceph_manager.c1:Canceling any pending splits or merges... 2023-10-12T19:41:00.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-12T19:41:01.318 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-12T19:41:01.318 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":60,"fsid":"1d86b02b-9ae0-4b5a-846b-2e446987469a","created":"2023-10-12T18:45:40.082660+0000","modified":"2023-10-12T19:30:03.849773+0000","last_up_change":"2023-10-12T18:45:46.501810+0000","last_in_change":"2023-10-12T18:45:41.050780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-10-12T18:45:46.539390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T18:45:48.714939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-10-12T18:46:24.697628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-10-12T18:46:26.433051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-10-12T18:46:36.340353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-10-12T18:46:38.173681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-10-12T18:46:40.161124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-10-12T18:46:42.184203+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"a1.rgw.buckets.data","create_time":"2023-10-12T18:46:56.462009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"a1.rgw.log","create_time":"2023-10-12T18:46:58.342011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"a1.rgw.control","create_time":"2023-10-12T18:47:00.203312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"a1.rgw.meta","create_time":"2023-10-12T18:47:02.233881+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"a2.rgw.buckets.data","create_time":"2023-10-12T18:47:12.928841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"a2.rgw.log","create_time":"2023-10-12T18:47:14.424433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"a2.rgw.control","create_time":"2023-10-12T18:47:16.244391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"a2.rgw.meta","create_time":"2023-10-12T18:47:18.272942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"a1.rgw.otp","create_time":"2023-10-12T18:47:27.837770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"a1.rgw.buckets.index","create_time":"2023-10-12T18:49:59.464844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"a2.rgw.buckets.index","create_time":"2023-10-12T18:50:02.219421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"a1.rgw.buckets.non-ec","create_time":"2023-10-12T19:30:01.792397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"7b28e758-0399-4d53-a70f-db692b665257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86172},{"type":"v1","addr":"172.21.15.170:6801","nonce":86172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86172},{"type":"v1","addr":"172.21.15.170:6803","nonce":86172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86172},{"type":"v1","addr":"172.21.15.170:6807","nonce":86172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86172},{"type":"v1","addr":"172.21.15.170:6805","nonce":86172}]},"public_addr":"172.21.15.170:6801/86172","cluster_addr":"172.21.15.170:6803/86172","heartbeat_back_addr":"172.21.15.170:6807/86172","heartbeat_front_addr":"172.21.15.170:6805/86172","state":["exists","up"]},{"osd":1,"uuid":"42fabdc9-4454-4f08-b147-fd3b5c276f02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86187},{"type":"v1","addr":"172.21.15.170:6809","nonce":86187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86187},{"type":"v1","addr":"172.21.15.170:6811","nonce":86187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86187},{"type":"v1","addr":"172.21.15.170:6816","nonce":86187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86187},{"type":"v1","addr":"172.21.15.170:6813","nonce":86187}]},"public_addr":"172.21.15.170:6809/86187","cluster_addr":"172.21.15.170:6811/86187","heartbeat_back_addr":"172.21.15.170:6816/86187","heartbeat_front_addr":"172.21.15.170:6813/86187","state":["exists","up"]},{"osd":2,"uuid":"abf486db-9793-43f5-af5f-5d4fea2a1a1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6815","nonce":86188},{"type":"v1","addr":"172.21.15.170:6817","nonce":86188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86188},{"type":"v1","addr":"172.21.15.170:6819","nonce":86188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86188},{"type":"v1","addr":"172.21.15.170:6823","nonce":86188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86188},{"type":"v1","addr":"172.21.15.170:6821","nonce":86188}]},"public_addr":"172.21.15.170:6817/86188","cluster_addr":"172.21.15.170:6819/86188","heartbeat_back_addr":"172.21.15.170:6823/86188","heartbeat_front_addr":"172.21.15.170:6821/86188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.442456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.493754+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T18:45:45.397730+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.2d","mappings":[{"from":0,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":0}]},{"pgid":"4.32","mappings":[{"from":1,"to":0}]},{"pgid":"4.3c","mappings":[{"from":1,"to":2}]},{"pgid":"10.1d","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T19:41:01.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-10-12T19:41:01.631 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-12T19:41:01.634 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-12T19:41:01.635 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-10-12T19:41:01.635 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:41:01.695 INFO:tasks.ceph.c1.osd.0:Stopped 2023-10-12T19:41:01.696 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-10-12T19:41:01.696 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:41:01.767 INFO:tasks.ceph.c1.osd.1:Stopped 2023-10-12T19:41:01.768 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-10-12T19:41:01.768 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:41:01.827 INFO:tasks.ceph.c1.osd.2:Stopped 2023-10-12T19:41:01.827 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-12T19:41:01.828 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-10-12T19:41:01.828 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:41:01.850 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-10-12T19:41:01.851 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-12T19:41:01.851 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-10-12T19:41:01.852 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T19:41:01.909 INFO:tasks.ceph.c1.mon.a:Stopped 2023-10-12T19:41:01.909 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-12T19:41:01.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-12T19:41:01.983 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T19:41:01.983 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-10-12T19:41:02.798 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T19:41:02.799 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-10-12T19:41:02.930 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T19:41:02.930 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-10-12T19:41:03.065 INFO:tasks.ceph:Archiving mon data... 2023-10-12T19:41:03.065 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584/data/c1.mon.a.tgz 2023-10-12T19:41:03.066 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-10-12T19:41:03.092 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.8ULK7ZMa4A 2023-10-12T19:41:03.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.8ULK7ZMa4A 2023-10-12T19:41:03.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.8ULK7ZMa4A 2023-10-12T19:41:03.637 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.8ULK7ZMa4A is 2MB 2023-10-12T19:41:03.751 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.8ULK7ZMa4A 2023-10-12T19:41:03.767 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-12T19:41:03.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-12T19:41:03.808 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-12T19:41:03.836 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-12T19:41:03.837 DEBUG:teuthology.orchestra.run.smithi170:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-12T19:41:03.873 DEBUG:teuthology.orchestra.run.smithi187:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-12T19:41:03.897 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-12T19:41:03.901 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-12T19:41:03.974 INFO:tasks.ceph:Archiving crash dumps... 2023-10-12T19:41:03.976 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584/remote/smithi170/crash 2023-10-12T19:41:03.977 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-12T19:41:04.003 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584/remote/smithi187/crash 2023-10-12T19:41:04.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-12T19:41:04.039 INFO:tasks.ceph:Compressing logs... 2023-10-12T19:41:04.040 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-12T19:41:04.048 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-12T19:41:04.179 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88007.log 2023-10-12T19:41:04.180 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.1.log 2023-10-12T19:41:04.180 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88007.log: gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.0.log 2023-10-12T19:41:04.180 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88007.log.gz 2023-10-12T19:41:04.181 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/rgw.c2.client.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88175.log 2023-10-12T19:41:04.181 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/rgw.c2.client.0.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.91933.log 2023-10-12T19:41:04.181 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88175.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.0.log 2023-10-12T19:41:04.181 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88175.log.gz 2023-10-12T19:41:04.182 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.91933.log: gzip -5 --verbose -- /var/log/ceph/c2.audit.log 2023-10-12T19:41:04.182 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.91933.log.gz 2023-10-12T19:41:04.182 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.90785.log 2023-10-12T19:41:04.183 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2.audit.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.95002.log 2023-10-12T19:41:04.183 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.90785.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.87923.log 2023-10-12T19:41:04.183 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.90785.log.gz 2023-10-12T19:41:04.183 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.91957.log 2023-10-12T19:41:04.184 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.95002.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.95002.log.gz 2023-10-12T19:41:04.184 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.87923.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.86276.log 2023-10-12T19:41:04.184 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.87923.log.gz 2023-10-12T19:41:04.184 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.91957.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88469.log 2023-10-12T19:41:04.185 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.91957.log.gz 2023-10-12T19:41:04.185 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.86276.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.92038.log 2023-10-12T19:41:04.185 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.86276.log.gz 2023-10-12T19:41:04.186 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88469.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88755.log 2023-10-12T19:41:04.186 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88469.log.gz 2023-10-12T19:41:04.186 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.92038.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.1.log 2023-10-12T19:41:04.186 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.92038.log.gz 2023-10-12T19:41:04.187 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88755.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88385.log 2023-10-12T19:41:04.187 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88755.log.gz 2023-10-12T19:41:04.187 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.86138.log 2023-10-12T19:41:04.188 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88385.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.90851.log 2023-10-12T19:41:04.188 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88385.log.gz 2023-10-12T19:41:04.188 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.86138.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.91891.log 2023-10-12T19:41:04.188 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.86138.log.gz 2023-10-12T19:41:04.189 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.90851.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.91794.log 2023-10-12T19:41:04.189 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.90851.log.gz 2023-10-12T19:41:04.189 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.91891.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.86348.log 2023-10-12T19:41:04.190 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.91891.log.gz 2023-10-12T19:41:04.190 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.91794.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.91712.log 2023-10-12T19:41:04.190 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.91794.log.gz 2023-10-12T19:41:04.190 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.86348.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88511.log 2023-10-12T19:41:04.191 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.86348.log.gz 2023-10-12T19:41:04.191 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.91712.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.2.log 2023-10-12T19:41:04.191 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.91712.log.gz 2023-10-12T19:41:04.191 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88511.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88301.log 2023-10-12T19:41:04.192 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88511.log.gz 2023-10-12T19:41:04.192 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.87881.log 2023-10-12T19:41:04.192 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88301.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88259.log 2023-10-12T19:41:04.193 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88301.log.gz 2023-10-12T19:41:04.193 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.87881.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88593.log 2023-10-12T19:41:04.193 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.87881.log.gz 2023-10-12T19:41:04.193 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88259.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.90827.log 2023-10-12T19:41:04.194 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88259.log.gz 2023-10-12T19:41:04.194 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88593.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.88958.log 2023-10-12T19:41:04.194 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88593.log.gz 2023-10-12T19:41:04.195 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.90827.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.87826.log 2023-10-12T19:41:04.195 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.90827.log.gz 2023-10-12T19:41:04.195 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.88958.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.93031.log 2023-10-12T19:41:04.195 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.87826.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88633.log 2023-10-12T19:41:04.196 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.87826.log.gz 2023-10-12T19:41:04.196 INFO:teuthology.orchestra.run.smithi187.stderr:gzip /var/log/ceph/c2-client.0.93031.log: -5 --verbose -- /var/log/ceph/c2-client.admin.88835.log 2023-10-12T19:41:04.196 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.93031.log.gz 2023-10-12T19:41:04.197 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88633.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88133.log 2023-10-12T19:41:04.197 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88633.log.gz 2023-10-12T19:41:04.197 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88835.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88835.log.gz 2023-10-12T19:41:04.197 INFO:teuthology.orchestra.run.smithi187.stderr: 76.6%gzip -5 -- replaced with /var/log/ceph/c2-client.0.88958.log.gz--verbose 2023-10-12T19:41:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:-- /var/log/ceph/c2-client.admin.85938.log 2023-10-12T19:41:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88427.log 2023-10-12T19:41:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88133.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88133.log.gz 2023-10-12T19:41:04.199 INFO:teuthology.orchestra.run.smithi187.stderr: 89.5% -- replaced with /var/log/ceph/c2.audit.log.gz 2023-10-12T19:41:04.199 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.85938.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.86096.log 2023-10-12T19:41:04.199 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.85938.log.gz 2023-10-12T19:41:04.199 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88427.log: gzip -5 --verbose -- /var/log/ceph/c2-mon.a.log 2023-10-12T19:41:04.200 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88427.log.gz 2023-10-12T19:41:04.200 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.86096.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.87965.log 2023-10-12T19:41:04.200 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.86096.log.gz 2023-10-12T19:41:04.200 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.92937.log 2023-10-12T19:41:04.201 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.87965.log: gzip -5 --verbose -- /var/log/ceph/c2.log 2023-10-12T19:41:04.201 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.87965.log.gz 2023-10-12T19:41:04.201 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.92937.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88091.log 2023-10-12T19:41:04.202 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.92937.log.gz 2023-10-12T19:41:04.202 INFO:teuthology.orchestra.run.smithi187.stderr:gzip /var/log/ceph/c2.log: -5 --verbose -- /var/log/ceph/c2-client.admin.86178.log 2023-10-12T19:41:04.202 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88091.log: gzip -5 --verbose -- /var/log/ceph/c2-client.1.89130.log 2023-10-12T19:41:04.202 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88091.log.gz 2023-10-12T19:41:04.203 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.86178.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.88859.log 2023-10-12T19:41:04.203 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.86178.log.gz 2023-10-12T19:41:04.203 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.1.89130.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88795.log 2023-10-12T19:41:04.204 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.88859.log: gzip -5 --verbose -- /var/log/ceph/c2-mgr.x.log 2023-10-12T19:41:04.204 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88795.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.94962.log 2023-10-12T19:41:04.204 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88795.log.gz 2023-10-12T19:41:04.204 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88551.log 2023-10-12T19:41:04.205 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.94962.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.93125.log 2023-10-12T19:41:04.205 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.94962.log.gz 2023-10-12T19:41:04.205 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88551.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88715.log 2023-10-12T19:41:04.205 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88551.log.gz 2023-10-12T19:41:04.206 INFO:teuthology.orchestra.run.smithi187.stderr: 76.9% -- replaced with /var/log/ceph/c2-client.1.89130.log.gz 2023-10-12T19:41:04.206 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.93125.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.93125.log.gz 2023-10-12T19:41:04.206 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.90932.log 2023-10-12T19:41:04.207 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88715.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88715.log.gz 2023-10-12T19:41:04.207 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88217.log 2023-10-12T19:41:04.207 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.90932.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.93219.log 2023-10-12T19:41:04.207 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.90932.log.gz 2023-10-12T19:41:04.208 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88217.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.90688.log 2023-10-12T19:41:04.208 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88217.log.gz 2023-10-12T19:41:04.208 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.93219.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.90525.log 2023-10-12T19:41:04.209 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.93219.log.gz 2023-10-12T19:41:04.209 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.90688.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88343.log 2023-10-12T19:41:04.209 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.90688.log.gz 2023-10-12T19:41:04.209 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.90525.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.82667.log 2023-10-12T19:41:04.210 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.90525.log.gz 2023-10-12T19:41:04.210 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88343.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.90607.log 2023-10-12T19:41:04.210 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88343.log.gz 2023-10-12T19:41:04.211 INFO:teuthology.orchestra.run.smithi187.stderr: 88.6% -- replaced with /var/log/ceph/c2.log.gz 2023-10-12T19:41:04.211 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph.tmp-client.admin.82667.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.82667.log.gz 2023-10-12T19:41:04.211 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88675.log 2023-10-12T19:41:04.211 INFO:teuthology.orchestra.run.smithi187.stderr: 83.8% -- replaced with /var/log/ceph/c2-client.0.88859.log.gz 2023-10-12T19:41:04.212 INFO:teuthology.orchestra.run.smithi187.stderr:gzip /var/log/ceph/c2-client.0.90607.log: -5 --verbose -- /var/log/ceph/c2-client.1.89039.log 2023-10-12T19:41:04.212 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.88675.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.88049.log 2023-10-12T19:41:04.212 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88675.log.gz 2023-10-12T19:41:04.213 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.1.89039.log: /var/log/ceph/c2-client.admin.88049.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.88049.log.gz 2023-10-12T19:41:04.213 INFO:teuthology.orchestra.run.smithi187.stderr: 1.8% -- replaced with /var/log/ceph/c2-client.0.90607.log.gz 2023-10-12T19:41:04.213 INFO:teuthology.orchestra.run.smithi187.stderr: 84.5% -- replaced with /var/log/ceph/c2-client.1.89039.log.gz 2023-10-12T19:41:04.347 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.146940.log 2023-10-12T19:41:04.348 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.147222.log 2023-10-12T19:41:04.348 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146940.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102104.log 2023-10-12T19:41:04.348 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.146940.log.gz 2023-10-12T19:41:04.348 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147222.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100985.log 2023-10-12T19:41:04.349 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147222.log.gz 2023-10-12T19:41:04.349 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102104.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135609.log 2023-10-12T19:41:04.349 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102104.log.gz 2023-10-12T19:41:04.350 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100985.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147888.log 2023-10-12T19:41:04.350 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135609.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115935.log 2023-10-12T19:41:04.350 INFO:teuthology.orchestra.run.smithi170.stderr: 60.6% -- replaced with /var/log/ceph/c1-client.0.100985.log.gz 2023-10-12T19:41:04.350 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135609.log.gz 2023-10-12T19:41:04.351 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147888.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141507.log 2023-10-12T19:41:04.351 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147888.log.gz 2023-10-12T19:41:04.351 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.115935.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91978.log 2023-10-12T19:41:04.352 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115935.log.gz 2023-10-12T19:41:04.352 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141507.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123357.log 2023-10-12T19:41:04.352 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141507.log.gz 2023-10-12T19:41:04.352 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.91978.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136270.log 2023-10-12T19:41:04.353 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91978.log.gz 2023-10-12T19:41:04.353 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123357.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.145495.log 2023-10-12T19:41:04.353 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123357.log.gz 2023-10-12T19:41:04.354 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136270.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116405.log 2023-10-12T19:41:04.354 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136270.log.gz 2023-10-12T19:41:04.354 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145495.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122111.log 2023-10-12T19:41:04.354 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.145495.log.gz 2023-10-12T19:41:04.355 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116405.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96053.log 2023-10-12T19:41:04.355 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116405.log.gz 2023-10-12T19:41:04.355 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122111.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110336.log 2023-10-12T19:41:04.356 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122111.log.gz 2023-10-12T19:41:04.356 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96053.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108441.log 2023-10-12T19:41:04.356 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96053.log.gz 2023-10-12T19:41:04.356 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110336.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155314.log 2023-10-12T19:41:04.357 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110336.log.gz 2023-10-12T19:41:04.357 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108441.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97547.log 2023-10-12T19:41:04.357 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108441.log.gz 2023-10-12T19:41:04.357 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155314.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104818.log 2023-10-12T19:41:04.358 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155314.log.gz 2023-10-12T19:41:04.358 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97547.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104912.log 2023-10-12T19:41:04.358 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104818.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149238.log 2023-10-12T19:41:04.359 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104818.log.gz 2023-10-12T19:41:04.359 INFO:teuthology.orchestra.run.smithi170.stderr: 56.1% -- replaced with /var/log/ceph/c1-client.0.97547.log.gz 2023-10-12T19:41:04.359 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104912.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141319.log 2023-10-12T19:41:04.360 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104912.log.gz 2023-10-12T19:41:04.360 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.149238.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130752.log 2023-10-12T19:41:04.360 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.149238.log.gz 2023-10-12T19:41:04.360 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141319.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116980.log 2023-10-12T19:41:04.361 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141319.log.gz 2023-10-12T19:41:04.361 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130752.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.85991.log 2023-10-12T19:41:04.361 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130752.log.gz 2023-10-12T19:41:04.361 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116980.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125835.log 2023-10-12T19:41:04.362 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116980.log.gz 2023-10-12T19:41:04.362 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.85991.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116123.log 2023-10-12T19:41:04.362 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.85991.log.gz 2023-10-12T19:41:04.363 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125835.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113020.log 2023-10-12T19:41:04.363 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.125835.log.gz 2023-10-12T19:41:04.363 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116123.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154655.log 2023-10-12T19:41:04.363 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116123.log.gz 2023-10-12T19:41:04.364 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113020.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141413.log 2023-10-12T19:41:04.364 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113020.log.gz 2023-10-12T19:41:04.364 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154655.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98857.log 2023-10-12T19:41:04.365 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154655.log.gz 2023-10-12T19:41:04.365 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141413.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.141413.log.gz 2023-10-12T19:41:04.365 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.122789.log 2023-10-12T19:41:04.365 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98857.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122507.log 2023-10-12T19:41:04.366 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122789.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104062.log 2023-10-12T19:41:04.366 INFO:teuthology.orchestra.run.smithi170.stderr: 58.9% -- replaced with /var/log/ceph/c1-client.0.98857.log.gz 2023-10-12T19:41:04.366 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122789.log.gz 2023-10-12T19:41:04.366 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122507.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.139319.log 2023-10-12T19:41:04.367 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122507.log.gz 2023-10-12T19:41:04.367 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104062.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152849.log 2023-10-12T19:41:04.367 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104062.log.gz 2023-10-12T19:41:04.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139319.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101076.log 2023-10-12T19:41:04.368 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139319.log.gz 2023-10-12T19:41:04.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152849.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123166.log 2023-10-12T19:41:04.368 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152849.log.gz 2023-10-12T19:41:04.369 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.101076.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148088.log 2023-10-12T19:41:04.369 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123166.log: 46.3% -- replaced with /var/log/ceph/c1-client.0.101076.log.gz 2023-10-12T19:41:04.369 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.144841.log 2023-10-12T19:41:04.370 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123166.log.gz 2023-10-12T19:41:04.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148088.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110713.log 2023-10-12T19:41:04.370 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148088.log.gz 2023-10-12T19:41:04.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144841.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107585.log 2023-10-12T19:41:04.371 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.144841.log.gz 2023-10-12T19:41:04.371 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110713.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/c1-client.0.110713.log.gz-- 2023-10-12T19:41:04.371 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143118.log 2023-10-12T19:41:04.372 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107585.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114730.log 2023-10-12T19:41:04.372 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107585.log.gz 2023-10-12T19:41:04.372 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143118.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113305.log 2023-10-12T19:41:04.373 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143118.log.gz 2023-10-12T19:41:04.373 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114730.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.114730.log.gz 2023-10-12T19:41:04.374 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.155699.log 2023-10-12T19:41:04.374 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113305.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129702.log 2023-10-12T19:41:04.374 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113305.log.gz 2023-10-12T19:41:04.374 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155699.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120222.log 2023-10-12T19:41:04.375 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155699.log.gz 2023-10-12T19:41:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129702.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.129702.log.gz-5 2023-10-12T19:41:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:--verbose -- /var/log/ceph/c1-client.0.132084.log 2023-10-12T19:41:04.376 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120222.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128566.log 2023-10-12T19:41:04.376 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120222.log.gz 2023-10-12T19:41:04.376 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132084.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131232.log 2023-10-12T19:41:04.376 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132084.log.gz 2023-10-12T19:41:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128566.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.90791.log 2023-10-12T19:41:04.377 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128566.log.gz 2023-10-12T19:41:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131232.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135225.log 2023-10-12T19:41:04.378 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131232.log.gz 2023-10-12T19:41:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.90791.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115457.log 2023-10-12T19:41:04.378 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.90791.log.gz 2023-10-12T19:41:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135225.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143599.log 2023-10-12T19:41:04.379 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135225.log.gz 2023-10-12T19:41:04.379 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.115457.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142073.log 2023-10-12T19:41:04.379 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115457.log.gz 2023-10-12T19:41:04.380 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143599.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114918.log 2023-10-12T19:41:04.380 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143599.log.gz 2023-10-12T19:41:04.380 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142073.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.127055.log 2023-10-12T19:41:04.380 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142073.log.gz 2023-10-12T19:41:04.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114918.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/c1-client.0.114918.log.gz-- 2023-10-12T19:41:04.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140741.log 2023-10-12T19:41:04.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.127055.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100706.log 2023-10-12T19:41:04.382 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.127055.log.gz 2023-10-12T19:41:04.382 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140741.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96332.log 2023-10-12T19:41:04.382 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.140741.log.gz 2023-10-12T19:41:04.382 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100706.log: gzip 0.0%-5 -- replaced with /var/log/ceph/c1-client.0.100706.log.gz--verbose 2023-10-12T19:41:04.383 INFO:teuthology.orchestra.run.smithi170.stderr:-- /var/log/ceph/c1-client.0.121452.log 2023-10-12T19:41:04.383 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96332.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96147.log 2023-10-12T19:41:04.383 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96332.log.gz 2023-10-12T19:41:04.384 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121452.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154750.log 2023-10-12T19:41:04.384 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121452.log.gz 2023-10-12T19:41:04.384 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96147.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.96147.log.gzgzip 2023-10-12T19:41:04.384 INFO:teuthology.orchestra.run.smithi170.stderr:-5 --verbose -- /var/log/ceph/c1-client.0.134272.log 2023-10-12T19:41:04.385 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154750.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135798.log 2023-10-12T19:41:04.385 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154750.log.gz 2023-10-12T19:41:04.385 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134272.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119553.log 2023-10-12T19:41:04.386 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134272.log.gz 2023-10-12T19:41:04.386 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135798.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88672.log 2023-10-12T19:41:04.386 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135798.log.gz 2023-10-12T19:41:04.386 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119553.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101919.log 2023-10-12T19:41:04.387 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119553.log.gz 2023-10-12T19:41:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88672.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135413.log 2023-10-12T19:41:04.387 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88672.log.gz 2023-10-12T19:41:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.101919.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155605.log 2023-10-12T19:41:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135413.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88112.log 2023-10-12T19:41:04.388 INFO:teuthology.orchestra.run.smithi170.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.101919.log.gz 2023-10-12T19:41:04.388 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135413.log.gz 2023-10-12T19:41:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155605.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116029.log 2023-10-12T19:41:04.389 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155605.log.gz 2023-10-12T19:41:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88112.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119458.log 2023-10-12T19:41:04.389 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88112.log.gz 2023-10-12T19:41:04.390 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116029.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123071.log 2023-10-12T19:41:04.390 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116029.log.gz 2023-10-12T19:41:04.390 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119458.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117837.log 2023-10-12T19:41:04.391 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119458.log.gz 2023-10-12T19:41:04.391 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123071.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137603.log 2023-10-12T19:41:04.391 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123071.log.gz 2023-10-12T19:41:04.391 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117837.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110525.log 2023-10-12T19:41:04.392 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117837.log.gz 2023-10-12T19:41:04.392 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137603.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150762.log 2023-10-12T19:41:04.392 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137603.log.gz 2023-10-12T19:41:04.392 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110525.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120786.log 2023-10-12T19:41:04.393 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110525.log.gz 2023-10-12T19:41:04.393 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150762.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142647.log 2023-10-12T19:41:04.393 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.150762.log.gz 2023-10-12T19:41:04.394 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120786.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95962.log 2023-10-12T19:41:04.394 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120786.log.gz 2023-10-12T19:41:04.394 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142647.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151615.log 2023-10-12T19:41:04.394 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142647.log.gz 2023-10-12T19:41:04.395 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95962.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114354.log 2023-10-12T19:41:04.395 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95962.log.gz 2023-10-12T19:41:04.395 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151615.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119647.log 2023-10-12T19:41:04.396 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151615.log.gz 2023-10-12T19:41:04.396 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114354.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106816.log 2023-10-12T19:41:04.396 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114354.log.gz 2023-10-12T19:41:04.396 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119647.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153799.log 2023-10-12T19:41:04.397 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119647.log.gz 2023-10-12T19:41:04.397 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106816.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.82571.log 2023-10-12T19:41:04.397 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106816.log.gz 2023-10-12T19:41:04.397 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153799.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153038.log 2023-10-12T19:41:04.398 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153799.log.gz 2023-10-12T19:41:04.398 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.tmp-client.admin.82571.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153988.log 2023-10-12T19:41:04.398 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.82571.log.gz 2023-10-12T19:41:04.399 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153038.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125074.log 2023-10-12T19:41:04.399 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153038.log.gz 2023-10-12T19:41:04.399 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153988.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141602.log 2023-10-12T19:41:04.399 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153988.log.gz 2023-10-12T19:41:04.400 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125074.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.125074.log.gz 2023-10-12T19:41:04.400 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.122017.log 2023-10-12T19:41:04.400 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141602.log: gzip -5 --verbose -- /var/log/ceph/c1-mon.a.log 2023-10-12T19:41:04.401 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141602.log.gz 2023-10-12T19:41:04.401 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122017.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88630.log 2023-10-12T19:41:04.401 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122017.log.gz 2023-10-12T19:41:04.401 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155220.log 2023-10-12T19:41:04.402 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88630.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95757.log 2023-10-12T19:41:04.402 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88630.log.gz 2023-10-12T19:41:04.402 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155220.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153226.log 2023-10-12T19:41:04.402 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155220.log.gz 2023-10-12T19:41:04.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95757.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147034.log 2023-10-12T19:41:04.403 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95757.log.gz 2023-10-12T19:41:04.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153226.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143693.log 2023-10-12T19:41:04.404 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153226.log.gz 2023-10-12T19:41:04.404 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147034.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.127616.log 2023-10-12T19:41:04.404 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147034.log.gz 2023-10-12T19:41:04.404 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143693.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.140177.log 2023-10-12T19:41:04.405 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143693.log.gz 2023-10-12T19:41:04.405 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.127616.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100056.log 2023-10-12T19:41:04.405 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.127616.log.gz 2023-10-12T19:41:04.406 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140177.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117262.log 2023-10-12T19:41:04.406 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.140177.log.gz 2023-10-12T19:41:04.406 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100056.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107193.log 2023-10-12T19:41:04.406 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117262.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130086.log 2023-10-12T19:41:04.407 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117262.log.gz 2023-10-12T19:41:04.407 INFO:teuthology.orchestra.run.smithi170.stderr: 47.1% -- replaced with /var/log/ceph/c1-client.0.100056.log.gz 2023-10-12T19:41:04.407 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107193.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134658.log 2023-10-12T19:41:04.408 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107193.log.gz 2023-10-12T19:41:04.408 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130086.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121829.log 2023-10-12T19:41:04.408 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130086.log.gz 2023-10-12T19:41:04.408 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134658.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129992.log 2023-10-12T19:41:04.409 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134658.log.gz 2023-10-12T19:41:04.409 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121829.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.121829.log.gz-5 2023-10-12T19:41:04.409 INFO:teuthology.orchestra.run.smithi170.stderr:--verbose -- /var/log/ceph/c1-client.0.133424.log 2023-10-12T19:41:04.410 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129992.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103303.log 2023-10-12T19:41:04.410 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129992.log.gz 2023-10-12T19:41:04.410 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133424.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106345.log 2023-10-12T19:41:04.410 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133424.log.gz 2023-10-12T19:41:04.411 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103303.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88280.log 2023-10-12T19:41:04.411 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106345.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153611.log 2023-10-12T19:41:04.411 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106345.log.gz 2023-10-12T19:41:04.411 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88280.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105300.log 2023-10-12T19:41:04.412 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88280.log.gz 2023-10-12T19:41:04.412 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153611.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152661.log 2023-10-12T19:41:04.412 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153611.log.gz 2023-10-12T19:41:04.413 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105300.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.132852.log 2023-10-12T19:41:04.413 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105300.log.gz 2023-10-12T19:41:04.413 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152661.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.144359.log 2023-10-12T19:41:04.413 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152661.log.gz 2023-10-12T19:41:04.414 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132852.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148464.log 2023-10-12T19:41:04.414 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132852.log.gz 2023-10-12T19:41:04.414 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144359.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.144163.log 2023-10-12T19:41:04.415 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.144359.log.gz 2023-10-12T19:41:04.415 INFO:teuthology.orchestra.run.smithi170.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.103303.log.gz 2023-10-12T19:41:04.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148464.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119082.log 2023-10-12T19:41:04.415 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148464.log.gz 2023-10-12T19:41:04.416 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144163.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113869.log 2023-10-12T19:41:04.416 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.144163.log.gz 2023-10-12T19:41:04.416 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119082.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133519.log 2023-10-12T19:41:04.416 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119082.log.gz 2023-10-12T19:41:04.417 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113869.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131034.log 2023-10-12T19:41:04.417 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113869.log.gz 2023-10-12T19:41:04.417 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133519.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101723.log 2023-10-12T19:41:04.418 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133519.log.gz 2023-10-12T19:41:04.418 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131034.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152086.log 2023-10-12T19:41:04.418 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131034.log.gz 2023-10-12T19:41:04.418 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.101723.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88548.log 2023-10-12T19:41:04.419 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101723.log.gz 2023-10-12T19:41:04.419 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152086.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99041.log 2023-10-12T19:41:04.419 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152086.log.gz 2023-10-12T19:41:04.420 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88548.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112716.log 2023-10-12T19:41:04.420 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88548.log.gz 2023-10-12T19:41:04.420 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99041.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134366.log 2023-10-12T19:41:04.420 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.112716.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.112716.log.gz 2023-10-12T19:41:04.421 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94813.log 2023-10-12T19:41:04.421 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134366.log: 53.2% -- replaced with /var/log/ceph/c1-client.0.99041.log.gz 2023-10-12T19:41:04.421 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.137226.log 2023-10-12T19:41:04.422 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134366.log.gz 2023-10-12T19:41:04.422 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94813.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118215.log 2023-10-12T19:41:04.422 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94813.log.gz 2023-10-12T19:41:04.422 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137226.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104630.log 2023-10-12T19:41:04.423 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137226.log.gz 2023-10-12T19:41:04.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118215.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99777.log 2023-10-12T19:41:04.423 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118215.log.gz 2023-10-12T19:41:04.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104630.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.140647.log 2023-10-12T19:41:04.424 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104630.log.gz 2023-10-12T19:41:04.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99777.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109008.log 2023-10-12T19:41:04.424 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99777.log.gz 2023-10-12T19:41:04.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140647.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138942.log 2023-10-12T19:41:04.425 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.140647.log.gz 2023-10-12T19:41:04.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109008.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115366.log 2023-10-12T19:41:04.425 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109008.log.gz 2023-10-12T19:41:04.426 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138942.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143409.log 2023-10-12T19:41:04.426 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138942.log.gz 2023-10-12T19:41:04.426 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.115366.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133709.log 2023-10-12T19:41:04.427 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115366.log.gz 2023-10-12T19:41:04.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143409.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103770.log 2023-10-12T19:41:04.427 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143409.log.gz 2023-10-12T19:41:04.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133709.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93481.log 2023-10-12T19:41:04.428 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133709.log.gz 2023-10-12T19:41:04.428 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103770.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123260.log 2023-10-12T19:41:04.428 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103770.log.gz 2023-10-12T19:41:04.428 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.93481.log: gzip -5 --verbose -- /var/log/ceph/c1.audit.log 2023-10-12T19:41:04.429 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93481.log.gz 2023-10-12T19:41:04.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123260.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142459.log 2023-10-12T19:41:04.429 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123260.log.gz 2023-10-12T19:41:04.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1.audit.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128378.log 2023-10-12T19:41:04.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142459.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150188.log 2023-10-12T19:41:04.430 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142459.log.gz 2023-10-12T19:41:04.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128378.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.132473.log 2023-10-12T19:41:04.431 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128378.log.gz 2023-10-12T19:41:04.431 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150188.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143787.log 2023-10-12T19:41:04.431 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.150188.log.gz 2023-10-12T19:41:04.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132473.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150668.log 2023-10-12T19:41:04.432 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132473.log.gz 2023-10-12T19:41:04.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143787.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103029.log 2023-10-12T19:41:04.432 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143787.log.gz 2023-10-12T19:41:04.433 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150668.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93865.log 2023-10-12T19:41:04.433 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.150668.log.gz 2023-10-12T19:41:04.433 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103029.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96796.log 2023-10-12T19:41:04.433 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.93865.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147410.log 2023-10-12T19:41:04.434 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93865.log.gz 2023-10-12T19:41:04.434 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96796.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134562.log 2023-10-12T19:41:04.434 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96796.log.gz 2023-10-12T19:41:04.435 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147410.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101354.log 2023-10-12T19:41:04.435 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147410.log.gz 2023-10-12T19:41:04.435 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134562.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128950.log 2023-10-12T19:41:04.435 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134562.log.gz 2023-10-12T19:41:04.436 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.101354.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.140365.log 2023-10-12T19:41:04.436 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101354.log.gz 2023-10-12T19:41:04.436 INFO:teuthology.orchestra.run.smithi170.stderr: 60.3% -- replaced with /var/log/ceph/c1-client.0.103029.log.gz 2023-10-12T19:41:04.437 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128950.log: gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.0.log 2023-10-12T19:41:04.437 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128950.log.gz 2023-10-12T19:41:04.437 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140365.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131895.log 2023-10-12T19:41:04.437 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.140365.log.gz 2023-10-12T19:41:04.438 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/rgw.c1.client.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107869.log 2023-10-12T19:41:04.438 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131895.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116311.log 2023-10-12T19:41:04.438 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131895.log.gz 2023-10-12T19:41:04.439 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107869.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94907.log 2023-10-12T19:41:04.439 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107869.log.gz 2023-10-12T19:41:04.439 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116311.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96890.log 2023-10-12T19:41:04.439 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116311.log.gz 2023-10-12T19:41:04.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94907.log: gzip 0.0%-5 -- replaced with /var/log/ceph/c1-client.0.94907.log.gz--verbose 2023-10-12T19:41:04.440 INFO:teuthology.orchestra.run.smithi170.stderr:-- /var/log/ceph/c1-client.admin.87944.log 2023-10-12T19:41:04.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96890.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123650.log 2023-10-12T19:41:04.441 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96890.log.gz 2023-10-12T19:41:04.441 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.87944.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143220.log 2023-10-12T19:41:04.441 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.87944.log.gz 2023-10-12T19:41:04.441 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123650.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125168.log 2023-10-12T19:41:04.442 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123650.log.gz 2023-10-12T19:41:04.442 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143220.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150480.log 2023-10-12T19:41:04.442 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143220.log.gz 2023-10-12T19:41:04.442 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125168.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154176.log 2023-10-12T19:41:04.443 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.125168.log.gz 2023-10-12T19:41:04.443 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150480.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107679.log 2023-10-12T19:41:04.443 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.150480.log.gz 2023-10-12T19:41:04.444 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154176.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108913.log 2023-10-12T19:41:04.444 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154176.log.gz 2023-10-12T19:41:04.444 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107679.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125741.log 2023-10-12T19:41:04.444 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107679.log.gz 2023-10-12T19:41:04.445 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108913.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131706.log 2023-10-12T19:41:04.445 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108913.log.gz 2023-10-12T19:41:04.445 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125741.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102567.log 2023-10-12T19:41:04.446 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.125741.log.gz 2023-10-12T19:41:04.446 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131706.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126587.log 2023-10-12T19:41:04.446 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131706.log.gz 2023-10-12T19:41:04.447 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102567.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.145029.log 2023-10-12T19:41:04.447 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126587.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108819.log 2023-10-12T19:41:04.447 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126587.log.gz 2023-10-12T19:41:04.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145029.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95189.log 2023-10-12T19:41:04.448 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.145029.log.gz 2023-10-12T19:41:04.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108819.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103487.log 2023-10-12T19:41:04.448 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108819.log.gz 2023-10-12T19:41:04.449 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95189.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102473.log 2023-10-12T19:41:04.449 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95189.log.gz 2023-10-12T19:41:04.449 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103487.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148849.log 2023-10-12T19:41:04.449 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103487.log.gz 2023-10-12T19:41:04.450 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102473.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.132757.log 2023-10-12T19:41:04.450 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148849.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.87776.log 2023-10-12T19:41:04.450 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148849.log.gz 2023-10-12T19:41:04.451 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132757.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.85833.log 2023-10-12T19:41:04.451 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132757.log.gz 2023-10-12T19:41:04.451 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.87776.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98198.log 2023-10-12T19:41:04.451 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.87776.log.gz 2023-10-12T19:41:04.452 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.85833.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123452.log 2023-10-12T19:41:04.452 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.85833.log.gz 2023-10-12T19:41:04.452 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98198.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.156170.log 2023-10-12T19:41:04.453 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123452.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113775.log 2023-10-12T19:41:04.453 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123452.log.gz 2023-10-12T19:41:04.453 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156170.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124214.log 2023-10-12T19:41:04.453 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.156170.log.gz 2023-10-12T19:41:04.454 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113775.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114448.log 2023-10-12T19:41:04.454 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113775.log.gz 2023-10-12T19:41:04.454 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124214.log: gzip 0.0%-5 -- replaced with /var/log/ceph/c1-client.0.124214.log.gz--verbose 2023-10-12T19:41:04.455 INFO:teuthology.orchestra.run.smithi170.stderr:-- /var/log/ceph/c1-client.0.119270.log 2023-10-12T19:41:04.455 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114448.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116886.log 2023-10-12T19:41:04.455 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114448.log.gz 2023-10-12T19:41:04.455 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119270.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118988.log 2023-10-12T19:41:04.456 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119270.log.gz 2023-10-12T19:41:04.456 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116886.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95471.log 2023-10-12T19:41:04.456 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116886.log.gz 2023-10-12T19:41:04.456 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118988.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.157182.log 2023-10-12T19:41:04.457 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118988.log.gz 2023-10-12T19:41:04.457 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95471.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155888.log 2023-10-12T19:41:04.457 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95471.log.gz 2023-10-12T19:41:04.458 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.157182.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88154.log 2023-10-12T19:41:04.458 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.157182.log.gz 2023-10-12T19:41:04.458 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155888.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95848.log 2023-10-12T19:41:04.458 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155888.log.gz 2023-10-12T19:41:04.459 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88154.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/c1-client.admin.88154.log.gz 2023-10-12T19:41:04.459 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.87986.log 2023-10-12T19:41:04.459 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95848.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93771.log 2023-10-12T19:41:04.460 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95848.log.gz 2023-10-12T19:41:04.460 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.87986.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120504.log 2023-10-12T19:41:04.460 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.87986.log.gz 2023-10-12T19:41:04.460 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.93771.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.156747.log 2023-10-12T19:41:04.461 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93771.log.gz 2023-10-12T19:41:04.461 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120504.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105771.log 2023-10-12T19:41:04.461 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120504.log.gz 2023-10-12T19:41:04.462 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156747.log: gzip -5 --verbose 0.0%-- /var/log/ceph/c1-client.0.135515.log -- replaced with /var/log/ceph/c1-client.0.156747.log.gz 2023-10-12T19:41:04.462 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-12T19:41:04.462 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105771.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116596.log 2023-10-12T19:41:04.462 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105771.log.gz 2023-10-12T19:41:04.463 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135515.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129514.log 2023-10-12T19:41:04.463 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135515.log.gz 2023-10-12T19:41:04.463 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116596.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.140835.log 2023-10-12T19:41:04.463 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116596.log.gz 2023-10-12T19:41:04.464 INFO:teuthology.orchestra.run.smithi170.stderr: 47.0% -- replaced with /var/log/ceph/c1-client.0.102567.log.gz 2023-10-12T19:41:04.464 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129514.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97267.log 2023-10-12T19:41:04.464 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129514.log.gz 2023-10-12T19:41:04.465 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140835.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106439.log 2023-10-12T19:41:04.465 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.140835.log.gz 2023-10-12T19:41:04.465 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97267.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.144935.log 2023-10-12T19:41:04.465 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97267.log.gz 2023-10-12T19:41:04.466 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106439.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124497.log 2023-10-12T19:41:04.466 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106439.log.gz 2023-10-12T19:41:04.466 INFO:teuthology.orchestra.run.smithi170.stderr: 60.6% -- replaced with /var/log/ceph/c1-client.0.102473.log.gz 2023-10-12T19:41:04.467 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144935.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153705.log 2023-10-12T19:41:04.467 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.144935.log.gz 2023-10-12T19:41:04.467 INFO:teuthology.orchestra.run.smithi170.stderr: 47.0%/var/log/ceph/c1-client.0.124497.log: -- replaced with /var/log/ceph/c1-client.0.98198.log.gz 2023-10-12T19:41:04.467 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91724.log 2023-10-12T19:41:04.468 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124497.log.gz 2023-10-12T19:41:04.468 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153705.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125639.log 2023-10-12T19:41:04.468 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153705.log.gz 2023-10-12T19:41:04.469 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.91724.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107099.log 2023-10-12T19:41:04.469 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91724.log.gz 2023-10-12T19:41:04.469 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125639.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108725.log 2023-10-12T19:41:04.469 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.125639.log.gz 2023-10-12T19:41:04.470 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107099.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109772.log 2023-10-12T19:41:04.470 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107099.log.gz 2023-10-12T19:41:04.470 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108725.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88238.log 2023-10-12T19:41:04.471 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108725.log.gz 2023-10-12T19:41:04.471 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109772.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.109772.log.gz 2023-10-12T19:41:04.471 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.107773.log 2023-10-12T19:41:04.471 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88238.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135131.log 2023-10-12T19:41:04.472 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88238.log.gz 2023-10-12T19:41:04.472 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107773.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129420.log 2023-10-12T19:41:04.472 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107773.log.gz 2023-10-12T19:41:04.472 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135131.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137132.log 2023-10-12T19:41:04.473 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135131.log.gz 2023-10-12T19:41:04.473 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129420.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118026.log 2023-10-12T19:41:04.473 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129420.log.gz 2023-10-12T19:41:04.474 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137132.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105583.log 2023-10-12T19:41:04.474 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137132.log.gz 2023-10-12T19:41:04.474 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118026.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98567.log 2023-10-12T19:41:04.474 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118026.log.gz 2023-10-12T19:41:04.475 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105583.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97821.log 2023-10-12T19:41:04.475 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105583.log.gz 2023-10-12T19:41:04.475 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98567.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98382.log 2023-10-12T19:41:04.476 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98567.log.gz 2023-10-12T19:41:04.476 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97821.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100891.log 2023-10-12T19:41:04.476 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97821.log.gz 2023-10-12T19:41:04.476 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98382.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151427.log 2023-10-12T19:41:04.477 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100891.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.140271.log 2023-10-12T19:41:04.477 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100891.log.gz 2023-10-12T19:41:04.477 INFO:teuthology.orchestra.run.smithi170.stderr: 60.7% -- replaced with /var/log/ceph/c1-client.0.98382.log.gz 2023-10-12T19:41:04.477 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151427.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142365.log 2023-10-12T19:41:04.478 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151427.log.gz 2023-10-12T19:41:04.478 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140271.log: gzip -5 --verbose -- 0.0%/var/log/ceph/c1-client.0.156264.log 2023-10-12T19:41:04.478 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/c1-client.0.140271.log.gz 2023-10-12T19:41:04.479 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142365.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111959.log 2023-10-12T19:41:04.479 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142365.log.gz 2023-10-12T19:41:04.479 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156264.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137509.log 2023-10-12T19:41:04.479 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.156264.log.gz 2023-10-12T19:41:04.480 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111959.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147128.log 2023-10-12T19:41:04.480 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111959.log.gz 2023-10-12T19:41:04.480 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137509.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/c1-client.0.137509.log.gz-- 2023-10-12T19:41:04.480 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140081.log 2023-10-12T19:41:04.481 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147128.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99871.log 2023-10-12T19:41:04.481 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147128.log.gz 2023-10-12T19:41:04.481 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140081.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99593.log 2023-10-12T19:41:04.482 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.140081.log.gz 2023-10-12T19:41:04.482 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99871.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.127136.log 2023-10-12T19:41:04.482 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99871.log.gz 2023-10-12T19:41:04.482 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99593.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138847.log 2023-10-12T19:41:04.483 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.127136.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150950.log 2023-10-12T19:41:04.483 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.127136.log.gz 2023-10-12T19:41:04.483 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138847.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138274.log 2023-10-12T19:41:04.483 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138847.log.gz 2023-10-12T19:41:04.484 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150950.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.150950.log.gz 2023-10-12T19:41:04.484 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94241.log 2023-10-12T19:41:04.484 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138274.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.138274.log.gz 2023-10-12T19:41:04.485 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.117932.log 2023-10-12T19:41:04.485 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94241.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94429.log 2023-10-12T19:41:04.485 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94241.log.gz 2023-10-12T19:41:04.485 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117932.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99686.log 2023-10-12T19:41:04.486 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117932.log.gz 2023-10-12T19:41:04.486 INFO:teuthology.orchestra.run.smithi170.stderr: 53.2% -- replaced with /var/log/ceph/c1-client.0.99593.log.gz 2023-10-12T19:41:04.486 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.131137.log 2023-10-12T19:41:04.486 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94429.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.94429.log.gz 2023-10-12T19:41:04.487 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99686.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106628.log 2023-10-12T19:41:04.487 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99686.log.gz 2023-10-12T19:41:04.487 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131137.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103120.log 2023-10-12T19:41:04.489 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131137.log.gz 2023-10-12T19:41:04.489 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106628.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154082.log 2023-10-12T19:41:04.490 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106628.log.gz 2023-10-12T19:41:04.490 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103120.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130846.log 2023-10-12T19:41:04.490 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154082.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130940.log 2023-10-12T19:41:04.490 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154082.log.gz 2023-10-12T19:41:04.491 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130846.log: gzip -5 --verbose -- /var/log/ceph/c1-mgr.x.log 2023-10-12T19:41:04.491 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130846.log.gz 2023-10-12T19:41:04.491 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130940.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98663.log 2023-10-12T19:41:04.492 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130940.log.gz 2023-10-12T19:41:04.492 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121546.log 2023-10-12T19:41:04.492 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98663.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101260.log 2023-10-12T19:41:04.493 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98663.log.gz 2023-10-12T19:41:04.493 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121546.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94147.log 2023-10-12T19:41:04.493 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121546.log.gz 2023-10-12T19:41:04.493 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.101260.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116217.log 2023-10-12T19:41:04.494 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101260.log.gz 2023-10-12T19:41:04.494 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94147.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.94147.log.gz 2023-10-12T19:41:04.494 INFO:teuthology.orchestra.run.smithi170.stderr:-5 --verbose -- /var/log/ceph/c1-client.0.149615.log 2023-10-12T19:41:04.494 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116217.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153894.log 2023-10-12T19:41:04.495 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116217.log.gz 2023-10-12T19:41:04.495 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.149615.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133990.log 2023-10-12T19:41:04.495 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.149615.log.gz 2023-10-12T19:41:04.496 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153894.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.153894.log.gz 2023-10-12T19:41:04.496 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.148559.log 2023-10-12T19:41:04.496 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133990.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102289.log 2023-10-12T19:41:04.496 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133990.log.gz 2023-10-12T19:41:04.497 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148559.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155126.log 2023-10-12T19:41:04.497 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148559.log.gz 2023-10-12T19:41:04.497 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102289.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141696.log 2023-10-12T19:41:04.498 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102289.log.gz 2023-10-12T19:41:04.498 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155126.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.155126.log.gz 2023-10-12T19:41:04.498 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1.log 2023-10-12T19:41:04.498 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141696.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.90833.log 2023-10-12T19:41:04.499 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141696.log.gz 2023-10-12T19:41:04.499 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.132663.log 2023-10-12T19:41:04.499 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.90833.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135892.log 2023-10-12T19:41:04.500 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.90833.log.gz 2023-10-12T19:41:04.500 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132663.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96611.log 2023-10-12T19:41:04.500 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132663.log.gz 2023-10-12T19:41:04.500 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135892.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.135892.log.gz 2023-10-12T19:41:04.501 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.145977.log 2023-10-12T19:41:04.501 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96611.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.145789.log 2023-10-12T19:41:04.501 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96611.log.gz 2023-10-12T19:41:04.502 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145977.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.145977.log.gz 2023-10-12T19:41:04.502 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92059.log 2023-10-12T19:41:04.502 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.130376.log 2023-10-12T19:41:04.502 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145789.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.145789.log.gz 2023-10-12T19:41:04.503 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.92059.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.92059.log.gz 2023-10-12T19:41:04.503 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.109102.log 2023-10-12T19:41:04.503 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.102383.log 2023-10-12T19:41:04.503 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130376.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.130376.log.gz 2023-10-12T19:41:04.504 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109102.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.109102.log.gz 2023-10-12T19:41:04.504 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.139696.log 2023-10-12T19:41:04.504 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.112905.log 2023-10-12T19:41:04.505 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102383.log: /var/log/ceph/c1-client.0.139696.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.139696.log.gz 2023-10-12T19:41:04.505 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.110917.log 2023-10-12T19:41:04.505 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose /var/log/ceph/c1-client.0.112905.log: -- /var/log/ceph/c1-client.0.155982.log 2023-10-12T19:41:04.505 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112905.log.gz 2023-10-12T19:41:04.506 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110917.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.146834.log 2023-10-12T19:41:04.506 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110917.log.gz 2023-10-12T19:41:04.506 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155982.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129890.log 2023-10-12T19:41:04.507 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155982.log.gz 2023-10-12T19:41:04.507 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146834.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.146834.log.gz 2023-10-12T19:41:04.507 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-osd.0.log 2023-10-12T19:41:04.507 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 /var/log/ceph/c1-client.0.129890.log: --verbose -- /var/log/ceph/c1-client.0.133236.log 2023-10-12T19:41:04.508 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129890.log.gz 2023-10-12T19:41:04.508 INFO:teuthology.orchestra.run.smithi170.stderr: 88.8% -- replaced with /var/log/ceph/c1.log.gz 2023-10-12T19:41:04.508 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88856.log 2023-10-12T19:41:04.508 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.99224.log 2023-10-12T19:41:04.509 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133236.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.133236.log.gz 2023-10-12T19:41:04.509 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.88856.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149905.log 2023-10-12T19:41:04.509 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99224.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141884.log 2023-10-12T19:41:04.510 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.149905.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151052.log 0.0% 2023-10-12T19:41:04.510 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/c1-client.0.149905.log.gz 2023-10-12T19:41:04.510 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141884.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.139037.log 2023-10-12T19:41:04.510 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141884.log.gz 2023-10-12T19:41:04.511 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151052.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124792.log 2023-10-12T19:41:04.511 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151052.log.gz 2023-10-12T19:41:04.511 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139037.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154467.log 2023-10-12T19:41:04.512 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139037.log.gz 2023-10-12T19:41:04.512 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124792.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114259.log 2023-10-12T19:41:04.512 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124792.log.gz 2023-10-12T19:41:04.512 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154467.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.91912.log 2023-10-12T19:41:04.513 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154467.log.gz 2023-10-12T19:41:04.513 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114259.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138086.log 2023-10-12T19:41:04.513 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114259.log.gz 2023-10-12T19:41:04.513 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.91912.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148653.log 2023-10-12T19:41:04.514 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.91912.log.gz 2023-10-12T19:41:04.514 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138086.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110619.log 2023-10-12T19:41:04.514 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138086.log.gz 2023-10-12T19:41:04.515 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148653.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121357.log 2023-10-12T19:41:04.515 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148653.log.gz 2023-10-12T19:41:04.515 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110619.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112526.log 2023-10-12T19:41:04.515 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110619.log.gz 2023-10-12T19:41:04.516 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121357.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100797.log 2023-10-12T19:41:04.516 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121357.log.gz 2023-10-12T19:41:04.516 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.112526.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129138.log 2023-10-12T19:41:04.517 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112526.log.gz 2023-10-12T19:41:04.517 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100797.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106534.log 2023-10-12T19:41:04.517 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100797.log.gz 2023-10-12T19:41:04.517 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129138.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.86033.log 2023-10-12T19:41:04.518 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129138.log.gz 2023-10-12T19:41:04.518 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106534.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150292.log 2023-10-12T19:41:04.518 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106534.log.gz 2023-10-12T19:41:04.518 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.86033.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.86033.log.gz 2023-10-12T19:41:04.519 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.120974.log 2023-10-12T19:41:04.519 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150292.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.150292.log.gz 2023-10-12T19:41:04.519 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.152943.log 2023-10-12T19:41:04.520 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120974.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114161.log 2023-10-12T19:41:04.520 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120974.log.gz 2023-10-12T19:41:04.520 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152943.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126880.log 2023-10-12T19:41:04.520 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152943.log.gz 2023-10-12T19:41:04.521 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114161.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111669.log 2023-10-12T19:41:04.521 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114161.log.gz 2023-10-12T19:41:04.521 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126880.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.146740.log 2023-10-12T19:41:04.522 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126880.log.gz 2023-10-12T19:41:04.522 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111669.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95663.log 2023-10-12T19:41:04.522 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111669.log.gz 2023-10-12T19:41:04.522 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146740.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88364.log 2023-10-12T19:41:04.523 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.146740.log.gz 2023-10-12T19:41:04.523 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95663.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90938.log 2023-10-12T19:41:04.523 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95663.log.gz 2023-10-12T19:41:04.524 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88364.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117546.log 2023-10-12T19:41:04.524 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88364.log.gz 2023-10-12T19:41:04.524 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.90938.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137799.log 2023-10-12T19:41:04.524 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90938.log.gz 2023-10-12T19:41:04.525 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117546.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155510.log 2023-10-12T19:41:04.525 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117546.log.gz 2023-10-12T19:41:04.525 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137799.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.132946.log 2023-10-12T19:41:04.525 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137799.log.gz 2023-10-12T19:41:04.526 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155510.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.156946.log 2023-10-12T19:41:04.526 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155510.log.gz 2023-10-12T19:41:04.526 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132946.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98291.log 2023-10-12T19:41:04.527 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132946.log.gz 2023-10-12T19:41:04.527 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156946.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.139884.log 2023-10-12T19:41:04.527 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.156946.log.gz 2023-10-12T19:41:04.527 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98291.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.144653.log 2023-10-12T19:41:04.528 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139884.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155793.log 2023-10-12T19:41:04.528 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139884.log.gz 2023-10-12T19:41:04.528 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144653.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/c1-client.0.144653.log.gz/var/log/ceph/c1-client.0.120128.log 2023-10-12T19:41:04.529 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-12T19:41:04.529 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155793.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88955.log 2023-10-12T19:41:04.529 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155793.log.gz 2023-10-12T19:41:04.529 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120128.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.139225.log 2023-10-12T19:41:04.530 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120128.log.gz 2023-10-12T19:41:04.530 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.88955.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111105.log 2023-10-12T19:41:04.530 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139225.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105100.log 2023-10-12T19:41:04.531 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139225.log.gz 2023-10-12T19:41:04.531 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111105.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88792.log 2023-10-12T19:41:04.531 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111105.log.gz 2023-10-12T19:41:04.531 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105100.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96426.log 2023-10-12T19:41:04.532 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105100.log.gz 2023-10-12T19:41:04.532 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88792.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113399.log 2023-10-12T19:41:04.532 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88792.log.gz 2023-10-12T19:41:04.532 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96426.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94335.log 2023-10-12T19:41:04.533 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96426.log.gz 2023-10-12T19:41:04.533 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113399.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149803.log 2023-10-12T19:41:04.533 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113399.log.gz 2023-10-12T19:41:04.534 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94335.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94523.log 2023-10-12T19:41:04.534 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94335.log.gz 2023-10-12T19:41:04.534 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.149803.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109197.log 2023-10-12T19:41:04.534 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.149803.log.gz 2023-10-12T19:41:04.535 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94523.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128856.log 2023-10-12T19:41:04.535 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94523.log.gz 2023-10-12T19:41:04.535 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109197.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99134.log 2023-10-12T19:41:04.535 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109197.log.gz 2023-10-12T19:41:04.536 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128856.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101632.log 2023-10-12T19:41:04.536 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128856.log.gz 2023-10-12T19:41:04.536 INFO:teuthology.orchestra.run.smithi170.stderr: 47.1% -- replaced with /var/log/ceph/c1-client.0.103120.log.gz 2023-10-12T19:41:04.537 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.139986.log 2023-10-12T19:41:04.537 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99134.log: /var/log/ceph/c1-client.0.101632.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129608.log 2023-10-12T19:41:04.537 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101632.log.gz 2023-10-12T19:41:04.537 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139986.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.146165.log 2023-10-12T19:41:04.538 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139986.log.gz 2023-10-12T19:41:04.538 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129608.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99503.log 2023-10-12T19:41:04.538 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129608.log.gz 2023-10-12T19:41:04.539 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146165.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151146.log 2023-10-12T19:41:04.539 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.146165.log.gz 2023-10-12T19:41:04.539 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99503.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126681.log 2023-10-12T19:41:04.539 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151146.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135704.log 2023-10-12T19:41:04.540 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151146.log.gz 2023-10-12T19:41:04.540 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126681.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96241.log 2023-10-12T19:41:04.540 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126681.log.gz 2023-10-12T19:41:04.540 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135704.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97453.log 2023-10-12T19:41:04.541 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135704.log.gz 2023-10-12T19:41:04.541 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96241.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111293.log 2023-10-12T19:41:04.541 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96241.log.gz 2023-10-12T19:41:04.542 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97453.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128284.log 2023-10-12T19:41:04.542 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97453.log.gz 2023-10-12T19:41:04.542 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.121923.log 2023-10-12T19:41:04.542 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111293.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.111293.log.gz 2023-10-12T19:41:04.543 INFO:teuthology.orchestra.run.smithi170.stderr: 46.3% -- replaced with /var/log/ceph/c1-client.0.102383.log.gz 2023-10-12T19:41:04.543 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128284.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136082.log 2023-10-12T19:41:04.543 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128284.log.gz 2023-10-12T19:41:04.544 INFO:teuthology.orchestra.run.smithi170.stderr: 84.0% -- replaced with /var/log/ceph/c1-client.0.88856.log.gz 2023-10-12T19:41:04.544 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.136850.log 2023-10-12T19:41:04.544 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121923.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.121923.log.gz 2023-10-12T19:41:04.545 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136082.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.139413.log 2023-10-12T19:41:04.545 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136082.log.gz 2023-10-12T19:41:04.545 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136850.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96984.log 2023-10-12T19:41:04.545 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136850.log.gz 2023-10-12T19:41:04.546 INFO:teuthology.orchestra.run.smithi170.stderr: 53.2% -- replaced with /var/log/ceph/c1-client.0.99224.log.gz 2023-10-12T19:41:04.546 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139413.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118120.log 2023-10-12T19:41:04.546 INFO:teuthology.orchestra.run.smithi170.stderr: 60.9% -- replaced with /var/log/ceph/c1-client.0.98291.log.gz 2023-10-12T19:41:04.547 INFO:teuthology.orchestra.run.smithi170.stderr: 76.9% -- replaced with /var/log/ceph/c1-client.0.88955.log.gz 2023-10-12T19:41:04.547 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139413.log.gz 2023-10-12T19:41:04.547 INFO:teuthology.orchestra.run.smithi170.stderr: 53.3% -- replaced with /var/log/ceph/c1-client.0.99503.log.gz 2023-10-12T19:41:04.547 INFO:teuthology.orchestra.run.smithi170.stderr: 53.6% -- replaced with /var/log/ceph/c1-client.0.99134.log.gz 2023-10-12T19:41:04.548 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96984.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134846.log 2023-10-12T19:41:04.548 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96984.log.gz 2023-10-12T19:41:04.548 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118120.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.118120.log.gz 2023-10-12T19:41:04.549 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.156556.log 2023-10-12T19:41:04.549 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134846.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117357.log 2023-10-12T19:41:04.549 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134846.log.gz 2023-10-12T19:41:04.549 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156556.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.139131.log 2023-10-12T19:41:04.550 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.156556.log.gz 2023-10-12T19:41:04.550 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117357.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125450.log 2023-10-12T19:41:04.550 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117357.log.gz 2023-10-12T19:41:04.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139131.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113116.log 2023-10-12T19:41:04.551 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139131.log.gz 2023-10-12T19:41:04.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125450.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95283.log 2023-10-12T19:41:04.551 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.125450.log.gz 2023-10-12T19:41:04.552 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113116.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112338.log 2023-10-12T19:41:04.552 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113116.log.gz 2023-10-12T19:41:04.552 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95283.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88406.log 2023-10-12T19:41:04.553 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95283.log.gz 2023-10-12T19:41:04.553 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.112338.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155032.log 2023-10-12T19:41:04.553 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112338.log.gz 2023-10-12T19:41:04.553 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88406.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149709.log 2023-10-12T19:41:04.554 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88406.log.gz 2023-10-12T19:41:04.554 INFO:teuthology.orchestra.run.smithi170.stderr: 89.1% -- replaced with /var/log/ceph/c1.audit.log.gz 2023-10-12T19:41:04.554 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155032.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153422.log 2023-10-12T19:41:04.555 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155032.log.gz 2023-10-12T19:41:04.555 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.149709.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125544.log 2023-10-12T19:41:04.555 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.149709.log.gz 2023-10-12T19:41:04.555 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153422.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99410.log 2023-10-12T19:41:04.556 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153422.log.gz 2023-10-12T19:41:04.556 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125544.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.146544.log 2023-10-12T19:41:04.556 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.125544.log.gz 2023-10-12T19:41:04.557 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99410.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152377.log 2023-10-12T19:41:04.557 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146544.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.140459.log 2023-10-12T19:41:04.557 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.146544.log.gz 2023-10-12T19:41:04.557 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152377.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111865.log 2023-10-12T19:41:04.558 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152377.log.gz 2023-10-12T19:41:04.558 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140459.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112620.log 2023-10-12T19:41:04.558 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.140459.log.gz 2023-10-12T19:41:04.559 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111865.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130184.log 2023-10-12T19:41:04.559 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111865.log.gz 2023-10-12T19:41:04.559 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.112620.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105865.log 2023-10-12T19:41:04.559 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112620.log.gz 2023-10-12T19:41:04.560 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130184.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.132280.log 2023-10-12T19:41:04.560 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130184.log.gz 2023-10-12T19:41:04.560 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105865.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.127313.log 2023-10-12T19:41:04.561 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105865.log.gz 2023-10-12T19:41:04.561 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132280.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147794.log 2023-10-12T19:41:04.561 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132280.log.gz 2023-10-12T19:41:04.561 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.127313.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105677.log 2023-10-12T19:41:04.562 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147794.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131423.log 2023-10-12T19:41:04.562 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147794.log.gz 2023-10-12T19:41:04.562 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105677.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120316.log 2023-10-12T19:41:04.563 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105677.log.gz 2023-10-12T19:41:04.563 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.98757.log 2023-10-12T19:41:04.563 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131423.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.131423.log.gz 2023-10-12T19:41:04.563 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120316.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136749.log 2023-10-12T19:41:04.564 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120316.log.gz 2023-10-12T19:41:04.564 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98757.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130658.log 2023-10-12T19:41:04.564 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136749.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115012.log 2023-10-12T19:41:04.565 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136749.log.gz 2023-10-12T19:41:04.565 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130658.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131800.log 2023-10-12T19:41:04.565 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130658.log.gz 2023-10-12T19:41:04.565 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.115012.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98014.log 2023-10-12T19:41:04.566 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115012.log.gz 2023-10-12T19:41:04.566 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131800.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104253.log 2023-10-12T19:41:04.566 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131800.log.gz 2023-10-12T19:41:04.567 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98014.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.144558.log 2023-10-12T19:41:04.567 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98014.log.gz 2023-10-12T19:41:04.567 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104253.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117168.log 2023-10-12T19:41:04.567 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104253.log.gz 2023-10-12T19:41:04.568 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144558.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90694.log 2023-10-12T19:41:04.568 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.144558.log.gz 2023-10-12T19:41:04.568 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117168.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105006.log 2023-10-12T19:41:04.568 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117168.log.gz 2023-10-12T19:41:04.569 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.90694.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106910.log 2023-10-12T19:41:04.569 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90694.log.gz 2023-10-12T19:41:04.569 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.121640.log 2023-10-12T19:41:04.570 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105006.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.105006.log.gz 2023-10-12T19:41:04.570 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106910.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118686.log 2023-10-12T19:41:04.570 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106910.log.gz 2023-10-12T19:41:04.570 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121640.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154373.log 2023-10-12T19:41:04.571 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121640.log.gz 2023-10-12T19:41:04.571 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118686.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114636.log 2023-10-12T19:41:04.571 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118686.log.gz 2023-10-12T19:41:04.572 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154373.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126305.log 2023-10-12T19:41:04.572 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154373.log.gz 2023-10-12T19:41:04.572 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114636.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154844.log 2023-10-12T19:41:04.572 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114636.log.gz 2023-10-12T19:41:04.573 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126305.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126117.log 2023-10-12T19:41:04.573 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126305.log.gz 2023-10-12T19:41:04.573 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154844.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111011.log 2023-10-12T19:41:04.574 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154844.log.gz 2023-10-12T19:41:04.574 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126117.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102841.log 2023-10-12T19:41:04.574 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126117.log.gz 2023-10-12T19:41:04.574 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111011.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147504.log 2023-10-12T19:41:04.575 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111011.log.gz 2023-10-12T19:41:04.575 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102841.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147598.log 2023-10-12T19:41:04.575 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102841.log.gz 2023-10-12T19:41:04.576 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147504.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151332.log 2023-10-12T19:41:04.576 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147504.log.gz 2023-10-12T19:41:04.576 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147598.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137320.log 2023-10-12T19:41:04.576 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147598.log.gz 2023-10-12T19:41:04.577 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151332.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151709.log 2023-10-12T19:41:04.577 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151332.log.gz 2023-10-12T19:41:04.577 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137320.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126778.log 2023-10-12T19:41:04.577 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137320.log.gz 2023-10-12T19:41:04.578 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151709.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103960.log 2023-10-12T19:41:04.578 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151709.log.gz 2023-10-12T19:41:04.578 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126778.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112149.log 2023-10-12T19:41:04.579 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126778.log.gz 2023-10-12T19:41:04.579 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103960.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102657.log 2023-10-12T19:41:04.579 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103960.log.gz 2023-10-12T19:41:04.579 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.112149.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.1.log 2023-10-12T19:41:04.580 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112149.log.gz 2023-10-12T19:41:04.580 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102657.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130470.log 2023-10-12T19:41:04.580 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153516.log 2023-10-12T19:41:04.581 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130470.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128096.log 2023-10-12T19:41:04.581 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130470.log.gz 2023-10-12T19:41:04.581 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 /var/log/ceph/c1-client.0.153516.log: --verbose -- /var/log/ceph/c1-client.0.145691.log 2023-10-12T19:41:04.581 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153516.log.gz 2023-10-12T19:41:04.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128096.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108245.log 2023-10-12T19:41:04.582 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128096.log.gz 2023-10-12T19:41:04.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145691.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133802.log 2023-10-12T19:41:04.583 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.145691.log.gz 2023-10-12T19:41:04.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108245.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104536.log 2023-10-12T19:41:04.583 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108245.log.gz 2023-10-12T19:41:04.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133802.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111575.log 2023-10-12T19:41:04.584 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133802.log.gz 2023-10-12T19:41:04.584 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104536.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126211.log 2023-10-12T19:41:04.584 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104536.log.gz 2023-10-12T19:41:04.585 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111575.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120692.log 2023-10-12T19:41:04.585 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111575.log.gz 2023-10-12T19:41:04.585 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126211.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.156076.log 2023-10-12T19:41:04.585 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126211.log.gz 2023-10-12T19:41:04.586 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120692.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90613.log 2023-10-12T19:41:04.586 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120692.log.gz 2023-10-12T19:41:04.586 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156076.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151992.log 2023-10-12T19:41:04.587 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.156076.log.gz 2023-10-12T19:41:04.587 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.90613.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93390.log 2023-10-12T19:41:04.587 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151992.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94053.log 2023-10-12T19:41:04.587 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151992.log.gz 2023-10-12T19:41:04.588 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.93390.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115653.log 2023-10-12T19:41:04.588 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93390.log.gz 2023-10-12T19:41:04.588 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94053.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98476.log 2023-10-12T19:41:04.588 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94053.log.gz 2023-10-12T19:41:04.589 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.115653.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117735.log 2023-10-12T19:41:04.589 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115653.log.gz 2023-10-12T19:41:04.589 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98476.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100154.log 2023-10-12T19:41:04.590 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98476.log.gz 2023-10-12T19:41:04.590 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117735.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128002.log 2023-10-12T19:41:04.590 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117735.log.gz 2023-10-12T19:41:04.590 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100154.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137705.log 2023-10-12T19:41:04.591 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128002.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.156841.log 2023-10-12T19:41:04.591 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128002.log.gz 2023-10-12T19:41:04.591 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137705.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99317.log 2023-10-12T19:41:04.592 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137705.log.gz 2023-10-12T19:41:04.592 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156841.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142930.log 2023-10-12T19:41:04.592 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.156841.log.gz 2023-10-12T19:41:04.592 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99317.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106722.log 2023-10-12T19:41:04.593 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142930.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152282.log 2023-10-12T19:41:04.593 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142930.log.gz 2023-10-12T19:41:04.593 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106722.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107390.log 2023-10-12T19:41:04.593 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106722.log.gz 2023-10-12T19:41:04.594 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152282.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149038.log 2023-10-12T19:41:04.594 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152282.log.gz 2023-10-12T19:41:04.594 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107390.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126974.log 2023-10-12T19:41:04.595 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107390.log.gz 2023-10-12T19:41:04.595 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.149038.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154270.log 0.0% 2023-10-12T19:41:04.595 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/c1-client.0.149038.log.gz 2023-10-12T19:41:04.595 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126974.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120410.log 2023-10-12T19:41:04.596 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126974.log.gz 2023-10-12T19:41:04.596 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154270.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152755.log 2023-10-12T19:41:04.596 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154270.log.gz 2023-10-12T19:41:04.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120410.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.144069.log 2023-10-12T19:41:04.597 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120410.log.gz 2023-10-12T19:41:04.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152755.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107005.log 2023-10-12T19:41:04.597 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152755.log.gz 2023-10-12T19:41:04.598 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144069.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153328.log 2023-10-12T19:41:04.598 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.144069.log.gz 2023-10-12T19:41:04.598 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107005.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102935.log 2023-10-12T19:41:04.599 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107005.log.gz 2023-10-12T19:41:04.599 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153328.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97730.log 2023-10-12T19:41:04.599 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153328.log.gz 2023-10-12T19:41:04.599 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102935.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119176.log 2023-10-12T19:41:04.600 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102935.log.gz 2023-10-12T19:41:04.600 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97730.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100429.log 2023-10-12T19:41:04.600 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97730.log.gz 2023-10-12T19:41:04.600 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119176.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113587.log 2023-10-12T19:41:04.601 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119176.log.gz 2023-10-12T19:41:04.601 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100429.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133896.log 2023-10-12T19:41:04.601 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100429.log.gz 2023-10-12T19:41:04.602 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113587.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147692.log 2023-10-12T19:41:04.602 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113587.log.gz 2023-10-12T19:41:04.602 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133896.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106053.log 2023-10-12T19:41:04.602 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133896.log.gz 2023-10-12T19:41:04.603 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147692.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117640.log 2023-10-12T19:41:04.603 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147692.log.gz 2023-10-12T19:41:04.603 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106053.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113211.log 2023-10-12T19:41:04.604 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106053.log.gz 2023-10-12T19:41:04.604 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117640.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141790.log 2023-10-12T19:41:04.604 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117640.log.gz 2023-10-12T19:41:04.604 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113211.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122412.log 2023-10-12T19:41:04.605 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113211.log.gz 2023-10-12T19:41:04.605 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141790.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121070.log 2023-10-12T19:41:04.605 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141790.log.gz 2023-10-12T19:41:04.605 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122412.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88028.log 2023-10-12T19:41:04.606 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122412.log.gz 2023-10-12T19:41:04.606 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121070.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102009.log 2023-10-12T19:41:04.606 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121070.log.gz 2023-10-12T19:41:04.607 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88028.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142742.log 2023-10-12T19:41:04.607 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88028.log.gz 2023-10-12T19:41:04.607 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102009.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122977.log 2023-10-12T19:41:04.607 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142742.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118498.log 2023-10-12T19:41:04.608 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142742.log.gz 2023-10-12T19:41:04.608 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122977.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121172.log 2023-10-12T19:41:04.608 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122977.log.gz 2023-10-12T19:41:04.609 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118498.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88196.log 2023-10-12T19:41:04.609 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118498.log.gz 2023-10-12T19:41:04.609 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121172.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.87860.log 2023-10-12T19:41:04.609 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121172.log.gz 2023-10-12T19:41:04.610 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88196.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141031.log 2023-10-12T19:41:04.610 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88196.log.gz 2023-10-12T19:41:04.610 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.87860.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141219.log 2023-10-12T19:41:04.610 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.87860.log.gz 2023-10-12T19:41:04.611 INFO:teuthology.orchestra.run.smithi170.stderr: 52.9% -- replaced with /var/log/ceph/c1-client.0.99410.log.gz 2023-10-12T19:41:04.611 INFO:teuthology.orchestra.run.smithi170.stderr: 4.8% -- replaced with /var/log/ceph/c1-client.0.127313.log.gz 2023-10-12T19:41:04.611 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141031.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138556.log 2023-10-12T19:41:04.612 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141031.log.gz 2023-10-12T19:41:04.612 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141219.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126493.log 2023-10-12T19:41:04.612 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141219.log.gz 2023-10-12T19:41:04.612 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138556.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117452.log 2023-10-12T19:41:04.613 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138556.log.gz 2023-10-12T19:41:04.613 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126493.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109481.log 2023-10-12T19:41:04.613 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126493.log.gz 2023-10-12T19:41:04.614 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117452.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/c1-client.0.117452.log.gz-- 2023-10-12T19:41:04.614 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98108.log 2023-10-12T19:41:04.614 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109481.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112811.log 2023-10-12T19:41:04.614 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109481.log.gz 2023-10-12T19:41:04.615 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98108.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123932.log 2023-10-12T19:41:04.615 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.112811.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148370.log 2023-10-12T19:41:04.615 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112811.log.gz 2023-10-12T19:41:04.615 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123932.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124309.log 2023-10-12T19:41:04.616 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123932.log.gz 2023-10-12T19:41:04.616 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148370.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.86244.log 2023-10-12T19:41:04.616 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148370.log.gz 2023-10-12T19:41:04.617 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124309.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97075.log 2023-10-12T19:41:04.617 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124309.log.gz 2023-10-12T19:41:04.617 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.86244.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111763.log 2023-10-12T19:41:04.617 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.86244.log.gz 2023-10-12T19:41:04.618 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97075.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.91954.log 2023-10-12T19:41:04.618 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97075.log.gz 2023-10-12T19:41:04.618 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111763.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154561.log 2023-10-12T19:41:04.619 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111763.log.gz 2023-10-12T19:41:04.619 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.91954.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151803.log 2023-10-12T19:41:04.619 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.91954.log.gz 2023-10-12T19:41:04.619 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154561.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101817.log 2023-10-12T19:41:04.620 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154561.log.gz 2023-10-12T19:41:04.620 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151803.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124697.log 2023-10-12T19:41:04.620 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151803.log.gz 2023-10-12T19:41:04.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.101817.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93299.log 2023-10-12T19:41:04.621 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101817.log.gz 2023-10-12T19:41:04.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124697.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114824.log 2023-10-12T19:41:04.621 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124697.log.gz 2023-10-12T19:41:04.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.93299.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108630.log 2023-10-12T19:41:04.622 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93299.log.gz 2023-10-12T19:41:04.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114824.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134752.log 2023-10-12T19:41:04.622 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114824.log.gz 2023-10-12T19:41:04.623 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108630.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.153132.log 2023-10-12T19:41:04.623 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108630.log.gz 2023-10-12T19:41:04.623 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134752.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.144747.log 2023-10-12T19:41:04.624 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134752.log.gz 2023-10-12T19:41:04.624 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.153132.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110242.log 2023-10-12T19:41:04.624 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153132.log.gz 2023-10-12T19:41:04.624 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144747.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131518.log 2023-10-12T19:41:04.625 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.144747.log.gz 2023-10-12T19:41:04.625 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110242.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105959.log 2023-10-12T19:41:04.625 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110242.log.gz 2023-10-12T19:41:04.625 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131518.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97915.log 2023-10-12T19:41:04.626 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131518.log.gz 2023-10-12T19:41:04.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105959.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93677.log 2023-10-12T19:41:04.626 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105959.log.gz 2023-10-12T19:41:04.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97915.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.157142.log 2023-10-12T19:41:04.627 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97915.log.gz 2023-10-12T19:41:04.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.93677.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.139510.log 2023-10-12T19:41:04.627 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93677.log.gz 2023-10-12T19:41:04.628 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.157142.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.146450.log 2023-10-12T19:41:04.628 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.157142.log.gz 2023-10-12T19:41:04.628 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139510.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124403.log 2023-10-12T19:41:04.629 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139510.log.gz 2023-10-12T19:41:04.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146450.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104159.log 2023-10-12T19:41:04.629 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.146450.log.gz 2023-10-12T19:41:04.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124403.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99965.log 2023-10-12T19:41:04.630 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124403.log.gz 2023-10-12T19:41:04.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104159.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.144453.log 2023-10-12T19:41:04.630 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104159.log.gz 2023-10-12T19:41:04.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.99965.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131990.log 2023-10-12T19:41:04.631 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144453.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112055.log 2023-10-12T19:41:04.631 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.144453.log.gz 2023-10-12T19:41:04.631 INFO:teuthology.orchestra.run.smithi170.stderr:gzip /var/log/ceph/c1-client.0.131990.log: -5 --verbose -- /var/log/ceph/c1-client.0.146356.log 2023-10-12T19:41:04.632 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131990.log.gz 2023-10-12T19:41:04.632 INFO:teuthology.orchestra.run.smithi170.stderr:gzip /var/log/ceph/c1-client.0.112055.log: -5 --verbose -- /var/log/ceph/c1-client.0.148747.log 2023-10-12T19:41:04.632 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112055.log.gz 2023-10-12T19:41:04.632 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146356.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.146356.log.gz 2023-10-12T19:41:04.633 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.125929.log 2023-10-12T19:41:04.633 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148747.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109960.log 2023-10-12T19:41:04.633 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148747.log.gz 2023-10-12T19:41:04.633 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125929.log: gzip -5 --verbose -- /var/log/ceph/c1-client.1.89036.log 2023-10-12T19:41:04.634 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.125929.log.gz 2023-10-12T19:41:04.634 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109960.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105202.log 2023-10-12T19:41:04.634 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109960.log.gz 2023-10-12T19:41:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.1.89036.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137898.log 2023-10-12T19:41:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105202.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.145217.log 2023-10-12T19:41:04.635 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105202.log.gz 2023-10-12T19:41:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137898.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149521.log 2023-10-12T19:41:04.636 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137898.log.gz 2023-10-12T19:41:04.636 INFO:teuthology.orchestra.run.smithi170.stderr: 59.5% -- replaced with /var/log/ceph/c1-client.0.98757.log.gz 2023-10-12T19:41:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145217.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.145123.log 2023-10-12T19:41:04.637 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.145217.log.gz 2023-10-12T19:41:04.637 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.149521.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.127911.log 2023-10-12T19:41:04.637 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.149521.log.gz 2023-10-12T19:41:04.637 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145123.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.2.log 2023-10-12T19:41:04.638 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.145123.log.gz 2023-10-12T19:41:04.638 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.127911.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109387.log 2023-10-12T19:41:04.638 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.127911.log.gz 2023-10-12T19:41:04.638 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101539.log 2023-10-12T19:41:04.639 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109387.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123554.log 2023-10-12T19:41:04.639 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109387.log.gz 2023-10-12T19:41:04.639 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.101539.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124886.log 2023-10-12T19:41:04.640 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123554.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151241.log 2023-10-12T19:41:04.640 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123554.log.gz 2023-10-12T19:41:04.640 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124886.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108339.log 2023-10-12T19:41:04.640 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124886.log.gz 2023-10-12T19:41:04.641 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151241.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128190.log 2023-10-12T19:41:04.641 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151241.log.gz 2023-10-12T19:41:04.641 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108339.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97637.log 2023-10-12T19:41:04.642 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108339.log.gz 2023-10-12T19:41:04.642 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128190.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120598.log 2023-10-12T19:41:04.642 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128190.log.gz 2023-10-12T19:41:04.642 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97637.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155408.log 2023-10-12T19:41:04.643 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120598.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150856.log 2023-10-12T19:41:04.643 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120598.log.gz 2023-10-12T19:41:04.643 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.155408.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136364.log 2023-10-12T19:41:04.644 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155408.log.gz 2023-10-12T19:41:04.644 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150856.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88322.log 2023-10-12T19:41:04.644 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.150856.log.gz 2023-10-12T19:41:04.644 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136364.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102195.log 2023-10-12T19:41:04.645 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136364.log.gz 2023-10-12T19:41:04.645 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88322.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136176.log 2023-10-12T19:41:04.645 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88322.log.gz 2023-10-12T19:41:04.645 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102195.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129044.log 2023-10-12T19:41:04.646 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102195.log.gz 2023-10-12T19:41:04.646 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136176.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.140553.log 2023-10-12T19:41:04.646 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136176.log.gz 2023-10-12T19:41:04.647 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129044.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100338.log 2023-10-12T19:41:04.647 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129044.log.gz 2023-10-12T19:41:04.647 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140553.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/c1-client.0.140553.log.gz-- 2023-10-12T19:41:04.647 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116499.log 2023-10-12T19:41:04.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100338.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121263.log 2023-10-12T19:41:04.648 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100338.log.gz 2023-10-12T19:41:04.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116499.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118781.log 2023-10-12T19:41:04.648 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116499.log.gz 2023-10-12T19:41:04.649 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121263.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115747.log 2023-10-12T19:41:04.649 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121263.log.gz 2023-10-12T19:41:04.649 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118781.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.118781.log.gz 2023-10-12T19:41:04.650 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90857.log 2023-10-12T19:41:04.650 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.115747.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103866.log 2023-10-12T19:41:04.650 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115747.log.gz 2023-10-12T19:41:04.650 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.90857.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137992.log 2023-10-12T19:41:04.651 INFO:teuthology.orchestra.run.smithi170.stderr: 56.9% -- replaced with /var/log/ceph/c1-client.0.97637.log.gz 2023-10-12T19:41:04.651 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90857.log.gz 2023-10-12T19:41:04.651 INFO:teuthology.orchestra.run.smithi170.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.102657.log.gz 2023-10-12T19:41:04.652 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103866.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88712.log 2023-10-12T19:41:04.652 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103866.log.gz 2023-10-12T19:41:04.652 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137992.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88070.log 2023-10-12T19:41:04.652 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137992.log.gz 2023-10-12T19:41:04.653 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88712.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138745.log 2023-10-12T19:41:04.653 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88712.log.gz 2023-10-12T19:41:04.653 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88070.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106148.log 0.0% 2023-10-12T19:41:04.653 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/c1-client.admin.88070.log.gz 2023-10-12T19:41:04.654 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138745.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104442.log 2023-10-12T19:41:04.654 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138745.log.gz 2023-10-12T19:41:04.654 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106148.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91815.log 2023-10-12T19:41:04.655 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106148.log.gz 2023-10-12T19:41:04.655 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104442.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138368.log 2023-10-12T19:41:04.655 INFO:teuthology.orchestra.run.smithi170.stderr: 47.1% -- replaced with /var/log/ceph/c1-client.0.101539.log.gz 2023-10-12T19:41:04.655 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104442.log.gz 2023-10-12T19:41:04.656 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.91815.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123744.log 2023-10-12T19:41:04.656 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91815.log.gz 2023-10-12T19:41:04.656 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138368.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102750.log 2023-10-12T19:41:04.656 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138368.log.gz 2023-10-12T19:41:04.657 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123744.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.145597.log 2023-10-12T19:41:04.657 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123744.log.gz 2023-10-12T19:41:04.657 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.102750.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116690.log 2023-10-12T19:41:04.658 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102750.log.gz 2023-10-12T19:41:04.658 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145597.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.146260.log 2023-10-12T19:41:04.658 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.145597.log.gz 2023-10-12T19:41:04.658 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116690.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131612.log 2023-10-12T19:41:04.659 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116690.log.gz 2023-10-12T19:41:04.659 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146260.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124026.log 2023-10-12T19:41:04.659 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.146260.log.gz 2023-10-12T19:41:04.660 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131612.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133613.log 2023-10-12T19:41:04.660 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131612.log.gz 2023-10-12T19:41:04.660 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124026.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122695.log 2023-10-12T19:41:04.660 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124026.log.gz 2023-10-12T19:41:04.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133613.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143315.log 2023-10-12T19:41:04.661 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133613.log.gz 2023-10-12T19:41:04.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122695.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135319.log 2023-10-12T19:41:04.662 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122695.log.gz 2023-10-12T19:41:04.662 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143315.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151521.log 2023-10-12T19:41:04.662 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143315.log.gz 2023-10-12T19:41:04.662 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135319.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109292.log 2023-10-12T19:41:04.663 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135319.log.gz 2023-10-12T19:41:04.663 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151521.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117074.log 2023-10-12T19:41:04.663 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.151521.log.gz 2023-10-12T19:41:04.664 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109292.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143505.log 2023-10-12T19:41:04.664 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109292.log.gz 2023-10-12T19:41:04.664 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.117074.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118310.log 2023-10-12T19:41:04.664 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117074.log.gz 2023-10-12T19:41:04.665 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143505.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121734.log 2023-10-12T19:41:04.665 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143505.log.gz 2023-10-12T19:41:04.665 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118310.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93583.log 2023-10-12T19:41:04.665 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118310.log.gz 2023-10-12T19:41:04.666 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.121734.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138180.log 2023-10-12T19:41:04.666 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121734.log.gz 2023-10-12T19:41:04.666 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.93583.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128660.log 2023-10-12T19:41:04.667 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93583.log.gz 2023-10-12T19:41:04.667 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138180.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150574.log 2023-10-12T19:41:04.667 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138180.log.gz 2023-10-12T19:41:04.667 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128660.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114542.log 2023-10-12T19:41:04.668 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128660.log.gz 2023-10-12T19:41:04.668 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150574.log: gzip -5 --verbose 0.0%-- /var/log/ceph/c1-client.0.135034.log -- replaced with /var/log/ceph/c1-client.0.150574.log.gz 2023-10-12T19:41:04.668 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-12T19:41:04.668 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114542.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109866.log 2023-10-12T19:41:04.669 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114542.log.gz 2023-10-12T19:41:04.669 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135034.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141125.log 2023-10-12T19:41:04.669 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135034.log.gz 2023-10-12T19:41:04.670 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109866.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131329.log 2023-10-12T19:41:04.670 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109866.log.gz 2023-10-12T19:41:04.670 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141125.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133330.log 2023-10-12T19:41:04.670 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141125.log.gz 2023-10-12T19:41:04.671 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.131329.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119741.log 2023-10-12T19:41:04.671 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131329.log.gz 2023-10-12T19:41:04.671 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133330.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.145883.log 2023-10-12T19:41:04.672 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133330.log.gz 2023-10-12T19:41:04.672 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119741.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.132568.log 2023-10-12T19:41:04.672 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119741.log.gz 2023-10-12T19:41:04.672 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145883.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100522.log 2023-10-12T19:41:04.673 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.145883.log.gz 2023-10-12T19:41:04.673 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132568.log: gzip -5 --verbose -- /var/log/ceph/c1-client.1.89127.log 2023-10-12T19:41:04.673 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132568.log.gz 2023-10-12T19:41:04.673 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100522.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.86171.log 2023-10-12T19:41:04.674 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100522.log.gz 2023-10-12T19:41:04.674 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.1.89127.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119364.log 2023-10-12T19:41:04.674 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.86171.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111481.log 2023-10-12T19:41:04.675 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.86171.log.gz 2023-10-12T19:41:04.675 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119364.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111199.log 2023-10-12T19:41:04.675 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119364.log.gz 2023-10-12T19:41:04.675 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111481.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119835.log 2023-10-12T19:41:04.676 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111481.log.gz 2023-10-12T19:41:04.676 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111199.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110148.log 2023-10-12T19:41:04.676 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111199.log.gz 2023-10-12T19:41:04.677 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119835.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150000.log 2023-10-12T19:41:04.677 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119835.log.gz 2023-10-12T19:41:04.677 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110148.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88590.log 2023-10-12T19:41:04.677 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110148.log.gz 2023-10-12T19:41:04.678 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150000.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95565.log 2023-10-12T19:41:04.678 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.150000.log.gz 2023-10-12T19:41:04.678 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88590.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.145316.log 2023-10-12T19:41:04.678 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88590.log.gz 2023-10-12T19:41:04.679 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95565.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.95565.log.gz 2023-10-12T19:41:04.679 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.124120.log 2023-10-12T19:41:04.679 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145316.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.145316.log.gz 2023-10-12T19:41:04.680 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.87721.log 2023-10-12T19:41:04.680 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124120.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138650.log 2023-10-12T19:41:04.680 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124120.log.gz 2023-10-12T19:41:04.680 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.87721.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135987.log 2023-10-12T19:41:04.681 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.87721.log.gz 2023-10-12T19:41:04.681 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138650.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136458.log 2023-10-12T19:41:04.681 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138650.log.gz 2023-10-12T19:41:04.682 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.135987.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.140929.log 2023-10-12T19:41:04.683 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135987.log.gz 2023-10-12T19:41:04.683 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136458.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110431.log 2023-10-12T19:41:04.683 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136458.log.gz 2023-10-12T19:41:04.683 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.140929.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.127491.log 2023-10-12T19:41:04.684 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.140929.log.gz 2023-10-12T19:41:04.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110431.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138462.log 2023-10-12T19:41:04.684 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110431.log.gz 2023-10-12T19:41:04.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.127491.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142836.log 2023-10-12T19:41:04.685 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.127491.log.gz 2023-10-12T19:41:04.685 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.138462.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97169.log 2023-10-12T19:41:04.685 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138462.log.gz 2023-10-12T19:41:04.686 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142836.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.139602.log 2023-10-12T19:41:04.686 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142836.log.gz 2023-10-12T19:41:04.686 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97169.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.156358.log 2023-10-12T19:41:04.686 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97169.log.gz 2023-10-12T19:41:04.687 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139602.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109575.log 2023-10-12T19:41:04.687 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139602.log.gz 2023-10-12T19:41:04.687 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156358.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142270.log 2023-10-12T19:41:04.688 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.156358.log.gz 2023-10-12T19:41:04.688 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109575.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.109575.log.gz 2023-10-12T19:41:04.688 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.119932.log 2023-10-12T19:41:04.688 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142270.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.139790.log 2023-10-12T19:41:04.689 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142270.log.gz 2023-10-12T19:41:04.689 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.119932.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94617.log 2023-10-12T19:41:04.689 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119932.log.gz 2023-10-12T19:41:04.689 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.139790.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101448.log 2023-10-12T19:41:04.690 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.139790.log.gz 2023-10-12T19:41:04.690 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94617.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149332.log 2023-10-12T19:41:04.690 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94617.log.gz 2023-10-12T19:41:04.691 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.101448.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115093.log 2023-10-12T19:41:04.691 INFO:teuthology.orchestra.run.smithi170.stderr:gzip /var/log/ceph/c1-client.0.149332.log: -5 --verbose -- /var/log/ceph/c1-client.0.146646.log 2023-10-12T19:41:04.691 INFO:teuthology.orchestra.run.smithi170.stderr: 77.2% -- replaced with /var/log/ceph/c1-client.1.89127.log.gz 2023-10-12T19:41:04.691 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.149332.log.gz 2023-10-12T19:41:04.693 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.115093.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115841.log 2023-10-12T19:41:04.693 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115093.log.gz 2023-10-12T19:41:04.694 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146646.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.146646.log.gz 2023-10-12T19:41:04.694 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.104348.log 2023-10-12T19:41:04.694 INFO:teuthology.orchestra.run.smithi170.stderr: 60.9% -- replaced with /var/log/ceph/c1-client.0.101448.log.gz 2023-10-12T19:41:04.694 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.115841.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126399.log 2023-10-12T19:41:04.695 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115841.log.gz 2023-10-12T19:41:04.695 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104348.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100245.log 2023-10-12T19:41:04.695 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104348.log.gz 2023-10-12T19:41:04.696 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126399.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124980.log 2023-10-12T19:41:04.696 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126399.log.gz 2023-10-12T19:41:04.696 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.100245.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129232.log 2023-10-12T19:41:04.696 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124980.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105394.log 2023-10-12T19:41:04.697 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124980.log.gz 2023-10-12T19:41:04.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129232.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152567.log 2023-10-12T19:41:04.697 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129232.log.gz 2023-10-12T19:41:04.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105394.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.105394.log.gz 2023-10-12T19:41:04.698 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.101169.log 2023-10-12T19:41:04.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152567.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97358.log 2023-10-12T19:41:04.698 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152567.log.gz 2023-10-12T19:41:04.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.101169.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128472.log 2023-10-12T19:41:04.699 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101169.log.gz 2023-10-12T19:41:04.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.97358.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108057.log 2023-10-12T19:41:04.699 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97358.log.gz 2023-10-12T19:41:04.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128472.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136654.log 2023-10-12T19:41:04.700 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128472.log.gz 2023-10-12T19:41:04.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108057.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120880.log 2023-10-12T19:41:04.701 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108057.log.gz 2023-10-12T19:41:04.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136654.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.129796.log 2023-10-12T19:41:04.701 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136654.log.gz 2023-10-12T19:41:04.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120880.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113963.log 2023-10-12T19:41:04.702 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120880.log.gz 2023-10-12T19:41:04.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129796.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116792.log 2023-10-12T19:41:04.702 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129796.log.gz 2023-10-12T19:41:04.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113963.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123838.log 2023-10-12T19:41:04.703 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113963.log.gz 2023-10-12T19:41:04.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.116792.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125262.log 2023-10-12T19:41:04.703 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116792.log.gz 2023-10-12T19:41:04.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.123838.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94719.log 2023-10-12T19:41:04.704 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123838.log.gz 2023-10-12T19:41:04.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125262.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.125262.log.gz 2023-10-12T19:41:04.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.120031.log 2023-10-12T19:41:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.94719.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149426.log 2023-10-12T19:41:04.705 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94719.log.gz 2023-10-12T19:41:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.120031.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133134.log 2023-10-12T19:41:04.705 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120031.log.gz 2023-10-12T19:41:04.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.149426.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118404.log 2023-10-12T19:41:04.706 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.149426.log.gz 2023-10-12T19:41:04.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133134.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96520.log 2023-10-12T19:41:04.707 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133134.log.gz 2023-10-12T19:41:04.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118404.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88752.log 2023-10-12T19:41:04.707 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118404.log.gz 2023-10-12T19:41:04.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96520.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107491.log 2023-10-12T19:41:04.708 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96520.log.gz 2023-10-12T19:41:04.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88752.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124603.log 2023-10-12T19:41:04.708 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88752.log.gz 2023-10-12T19:41:04.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107491.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122601.log 0.0% 2023-10-12T19:41:04.709 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/c1-client.0.107491.log.gz 2023-10-12T19:41:04.709 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.156651.log 2023-10-12T19:41:04.709 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.124603.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.124603.log.gz 2023-10-12T19:41:04.710 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122601.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141978.log 2023-10-12T19:41:04.710 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122601.log.gz 2023-10-12T19:41:04.710 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156651.log: 0.0%gzip -5 -- replaced with /var/log/ceph/c1-client.0.156651.log.gz--verbose 2023-10-12T19:41:04.710 INFO:teuthology.orchestra.run.smithi170.stderr:-- /var/log/ceph/c1-client.0.103676.log 2023-10-12T19:41:04.711 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.141978.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107288.log 2023-10-12T19:41:04.711 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141978.log.gz 2023-10-12T19:41:04.711 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103676.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122205.log 2023-10-12T19:41:04.711 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103676.log.gz 2023-10-12T19:41:04.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107288.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.107288.log.gz 2023-10-12T19:41:04.712 INFO:teuthology.orchestra.run.smithi170.stderr: 50.3% -- replaced with /var/log/ceph/c1-client.0.100245.log.gz 2023-10-12T19:41:04.712 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.151897.log 2023-10-12T19:41:04.713 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122205.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113681.log 2023-10-12T19:41:04.713 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122205.log.gz 2023-10-12T19:41:04.713 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.151897.log: gzip 0.0%-5 -- replaced with /var/log/ceph/c1-client.0.151897.log.gz--verbose 2023-10-12T19:41:04.713 INFO:teuthology.orchestra.run.smithi170.stderr:-- /var/log/ceph/c1-client.0.130281.log 2023-10-12T19:41:04.714 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.107963.log 2023-10-12T19:41:04.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113681.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.113681.log.gz 2023-10-12T19:41:04.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130281.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148943.log 2023-10-12T19:41:04.714 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130281.log.gz 2023-10-12T19:41:04.715 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.107963.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143975.log 2023-10-12T19:41:04.715 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107963.log.gz 2023-10-12T19:41:04.715 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.113493.log 2023-10-12T19:41:04.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148943.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.148943.log.gz 2023-10-12T19:41:04.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143975.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118592.log 2023-10-12T19:41:04.716 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143975.log.gz 2023-10-12T19:41:04.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.113493.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136552.log 2023-10-12T19:41:04.717 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113493.log.gz 2023-10-12T19:41:04.717 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.96705.log 2023-10-12T19:41:04.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118592.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.118592.log.gz 2023-10-12T19:41:04.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136552.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108151.log 2023-10-12T19:41:04.718 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136552.log.gz 2023-10-12T19:41:04.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.96705.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115551.log 2023-10-12T19:41:04.718 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96705.log.gz 2023-10-12T19:41:04.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108151.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154938.log 2023-10-12T19:41:04.719 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108151.log.gz 2023-10-12T19:41:04.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.115551.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148182.log 2023-10-12T19:41:04.719 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115551.log.gz 2023-10-12T19:41:04.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.154938.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150386.log 2023-10-12T19:41:04.720 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154938.log.gz 2023-10-12T19:41:04.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148182.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118884.log 2023-10-12T19:41:04.720 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148182.log.gz 2023-10-12T19:41:04.721 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150386.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110815.log 2023-10-12T19:41:04.721 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.150386.log.gz 2023-10-12T19:41:04.721 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.118884.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.132379.log 2023-10-12T19:41:04.722 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118884.log.gz 2023-10-12T19:41:04.722 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110815.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.145404.log 2023-10-12T19:41:04.722 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110815.log.gz 2023-10-12T19:41:04.722 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132379.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147993.log 2023-10-12T19:41:04.723 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132379.log.gz 2023-10-12T19:41:04.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.145404.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122310.log 2023-10-12T19:41:04.723 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.145404.log.gz 2023-10-12T19:41:04.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147993.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125356.log 2023-10-12T19:41:04.724 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147993.log.gz 2023-10-12T19:41:04.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122310.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.156452.log 2023-10-12T19:41:04.724 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122310.log.gz 2023-10-12T19:41:04.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.125356.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.125356.log.gzgzip 2023-10-12T19:41:04.725 INFO:teuthology.orchestra.run.smithi170.stderr:-5 --verbose -- /var/log/ceph/c1-client.0.149143.log 2023-10-12T19:41:04.725 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.105489.log 2023-10-12T19:41:04.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.156452.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.156452.log.gz 2023-10-12T19:41:04.726 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.149143.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133040.log 2023-10-12T19:41:04.726 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.149143.log.gz 2023-10-12T19:41:04.726 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.105489.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152188.log 2023-10-12T19:41:04.726 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105489.log.gz 2023-10-12T19:41:04.727 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.133040.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108535.log 2023-10-12T19:41:04.727 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133040.log.gz 2023-10-12T19:41:04.727 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152188.log: gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.1.log 2023-10-12T19:41:04.728 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152188.log.gz 2023-10-12T19:41:04.728 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.108535.log: gzip 0.0% -- replaced with /var/log/ceph/c1-client.0.108535.log.gz-5 2023-10-12T19:41:04.728 INFO:teuthology.orchestra.run.smithi170.stderr:--verbose -- /var/log/ceph/c1-client.0.112432.log 2023-10-12T19:41:04.728 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/rgw.c1.client.1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.146071.log 2023-10-12T19:41:04.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.112432.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.112432.log.gz 2023-10-12T19:41:04.729 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.109677.log 2023-10-12T19:41:04.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.146071.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.87902.log 2023-10-12T19:41:04.729 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.146071.log.gz 2023-10-12T19:41:04.730 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.109677.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.109677.log.gz 2023-10-12T19:41:04.730 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.111387.log 2023-10-12T19:41:04.730 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.87902.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.87818.log 2023-10-12T19:41:04.731 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.87902.log.gz 2023-10-12T19:41:04.731 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.111387.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.144257.log 2023-10-12T19:41:04.731 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111387.log.gz 2023-10-12T19:41:04.731 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.87818.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134178.log 2023-10-12T19:41:04.732 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.87818.log.gz 2023-10-12T19:41:04.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.144257.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142553.log 2023-10-12T19:41:04.732 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.144257.log.gz 2023-10-12T19:41:04.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134178.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98951.log 0.0% 2023-10-12T19:41:04.733 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/c1-client.0.134178.log.gz 2023-10-12T19:41:04.733 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142553.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136944.log 2023-10-12T19:41:04.733 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142553.log.gz 2023-10-12T19:41:04.734 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.98951.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.132186.log 2023-10-12T19:41:04.734 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.136944.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104724.log 2023-10-12T19:41:04.734 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136944.log.gz 2023-10-12T19:41:04.734 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.132186.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95095.log 2023-10-12T19:41:04.735 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132186.log.gz 2023-10-12T19:41:04.735 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.104724.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103582.log 2023-10-12T19:41:04.735 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104724.log.gz 2023-10-12T19:41:04.735 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95095.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.95095.log.gz 2023-10-12T19:41:04.736 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.106250.log 2023-10-12T19:41:04.736 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103582.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152471.log 2023-10-12T19:41:04.736 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103582.log.gz 2023-10-12T19:41:04.737 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.106250.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147316.log 2023-10-12T19:41:04.737 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106250.log.gz 2023-10-12T19:41:04.737 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.152471.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.88832.log 2023-10-12T19:41:04.737 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152471.log.gz 2023-10-12T19:41:04.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.147316.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.150094.log 2023-10-12T19:41:04.738 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147316.log.gz 2023-10-12T19:41:04.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.88832.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112243.log 2023-10-12T19:41:04.738 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.88832.log.gz 2023-10-12T19:41:04.739 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.150094.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.86073.log 2023-10-12T19:41:04.739 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.150094.log.gz 2023-10-12T19:41:04.739 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.112243.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134468.log 2023-10-12T19:41:04.740 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112243.log.gz 2023-10-12T19:41:04.740 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.admin.86073.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103213.log 2023-10-12T19:41:04.740 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.86073.log.gz 2023-10-12T19:41:04.740 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134468.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110054.log 2023-10-12T19:41:04.741 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134468.log.gz 2023-10-12T19:41:04.741 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103213.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143881.log 2023-10-12T19:41:04.741 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.110054.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93959.log 2023-10-12T19:41:04.741 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110054.log.gz 2023-10-12T19:41:04.742 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143881.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126023.log 2023-10-12T19:41:04.742 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143881.log.gz 2023-10-12T19:41:04.742 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.93959.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128754.log 2023-10-12T19:41:04.743 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93959.log.gz 2023-10-12T19:41:04.743 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.126023.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103396.log 2023-10-12T19:41:04.743 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.126023.log.gz 2023-10-12T19:41:04.743 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.128754.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114059.log 2023-10-12T19:41:04.744 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.128754.log.gz 2023-10-12T19:41:04.744 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.103396.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122883.log 2023-10-12T19:41:04.744 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103396.log.gz 2023-10-12T19:41:04.744 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.114059.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137038.log 2023-10-12T19:41:04.745 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114059.log.gz 2023-10-12T19:41:04.745 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.122883.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142175.log 0.0% 2023-10-12T19:41:04.745 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/c1-client.0.122883.log.gz 2023-10-12T19:41:04.745 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137038.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95377.log 2023-10-12T19:41:04.746 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137038.log.gz 2023-10-12T19:41:04.746 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.142175.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134940.log 2023-10-12T19:41:04.746 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142175.log.gz 2023-10-12T19:41:04.747 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95377.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.95377.log.gz 2023-10-12T19:41:04.747 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.100610.log 2023-10-12T19:41:04.747 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134940.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.127232.log 2023-10-12T19:41:04.747 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134940.log.gz 2023-10-12T19:41:04.748 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 /var/log/ceph/c1-client.0.100610.log: --verbose -- /var/log/ceph/c1-client.0.129326.log 2023-10-12T19:41:04.748 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100610.log.gz 2023-10-12T19:41:04.748 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.127232.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130564.log 2023-10-12T19:41:04.749 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.129326.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134084.log 2023-10-12T19:41:04.749 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129326.log.gz 2023-10-12T19:41:04.749 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.130564.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143024.log 2023-10-12T19:41:04.749 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130564.log.gz 2023-10-12T19:41:04.750 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.134084.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148276.log 2023-10-12T19:41:04.750 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134084.log.gz 2023-10-12T19:41:04.750 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.143024.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.137415.log 2023-10-12T19:41:04.750 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143024.log.gz 2023-10-12T19:41:04.751 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.148276.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95001.log 2023-10-12T19:41:04.751 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148276.log.gz 2023-10-12T19:41:04.751 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.137415.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90532.log 2023-10-12T19:41:04.752 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137415.log.gz 2023-10-12T19:41:04.752 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.95001.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.95001.log.gz 2023-10-12T19:41:04.752 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c1-client.0.90532.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.90532.log.gz 2023-10-12T19:41:04.752 INFO:teuthology.orchestra.run.smithi170.stderr: 52.4% -- replaced with /var/log/ceph/c1-client.0.98951.log.gz 2023-10-12T19:41:04.753 INFO:teuthology.orchestra.run.smithi170.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.103213.log.gz 2023-10-12T19:41:04.753 INFO:teuthology.orchestra.run.smithi170.stderr: 66.7% -- replaced with /var/log/ceph/c1-client.0.127232.log.gz 2023-10-12T19:41:04.753 INFO:teuthology.orchestra.run.smithi170.stderr: 84.5% 1.8% -- replaced with /var/log/ceph/c1-client.0.90613.log.gz -- replaced with /var/log/ceph/c1-client.1.89036.log.gz 2023-10-12T19:41:04.753 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-12T19:41:04.754 INFO:teuthology.orchestra.run.smithi170.stderr: 46.6% -- replaced with /var/log/ceph/c1-client.0.98108.log.gz 2023-10-12T19:41:04.754 INFO:teuthology.orchestra.run.smithi170.stderr: 65.7% -- replaced with /var/log/ceph/c1-client.0.99317.log.gz 2023-10-12T19:41:04.754 INFO:teuthology.orchestra.run.smithi170.stderr: 59.7% -- replaced with /var/log/ceph/c1-client.0.99965.log.gz 2023-10-12T19:41:04.754 INFO:teuthology.orchestra.run.smithi170.stderr: 63.0% -- replaced with /var/log/ceph/c1-client.0.100154.log.gz 2023-10-12T19:41:04.755 INFO:teuthology.orchestra.run.smithi170.stderr: 60.7% -- replaced with /var/log/ceph/c1-client.0.102009.log.gz 2023-10-12T19:41:05.322 INFO:teuthology.orchestra.run.smithi187.stderr: 93.2% -- replaced with /var/log/ceph/c2-mgr.x.log.gz 2023-10-12T19:41:06.916 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/c1-mgr.x.log.gz 2023-10-12T19:41:10.293 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/c1-mon.a.log.gz 2023-10-12T19:41:12.357 INFO:teuthology.orchestra.run.smithi187.stderr: 92.3% -- replaced with /var/log/ceph/c2-mon.a.log.gz 2023-10-12T19:42:36.789 INFO:teuthology.orchestra.run.smithi187.stderr: 94.1% -- replaced with /var/log/ceph/rgw.c2.client.1.log.gz 2023-10-12T19:42:36.842 INFO:teuthology.orchestra.run.smithi187.stderr: 94.2% -- replaced with /var/log/ceph/rgw.c2.client.0.log.gz 2023-10-12T19:43:20.879 INFO:teuthology.orchestra.run.smithi170.stderr: 94.1% -- replaced with /var/log/ceph/rgw.c1.client.1.log.gz 2023-10-12T19:46:06.603 INFO:teuthology.orchestra.run.smithi170.stderr: 94.1% -- replaced with /var/log/ceph/rgw.c1.client.0.log.gz 2023-10-12T19:54:29.074 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/c1-osd.1.log.gz 2023-10-12T19:56:00.299 INFO:teuthology.orchestra.run.smithi170.stderr: 93.7% -- replaced with /var/log/ceph/c1-osd.2.log.gz 2023-10-12T19:56:15.372 INFO:teuthology.orchestra.run.smithi170.stderr: 93.7% -- replaced with /var/log/ceph/c1-osd.0.log.gz 2023-10-12T19:56:15.406 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-12T19:56:15.406 INFO:teuthology.orchestra.run.smithi170.stderr:real 15m11.346s 2023-10-12T19:56:15.406 INFO:teuthology.orchestra.run.smithi170.stderr:user 16m57.801s 2023-10-12T19:56:15.407 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m45.162s 2023-10-12T19:59:42.195 INFO:teuthology.orchestra.run.smithi187.stderr: 93.6% -- replaced with /var/log/ceph/c2-osd.1.log.gz 2023-10-12T20:00:51.466 INFO:teuthology.orchestra.run.smithi187.stderr: 93.4% -- replaced with /var/log/ceph/c2-osd.2.log.gz 2023-10-12T20:03:54.423 INFO:teuthology.orchestra.run.smithi187.stderr: 93.6% -- replaced with /var/log/ceph/c2-osd.0.log.gz 2023-10-12T20:03:54.503 INFO:teuthology.orchestra.run.smithi187.stderr: 2023-10-12T20:03:54.504 INFO:teuthology.orchestra.run.smithi187.stderr:real 22m50.404s 2023-10-12T20:03:54.504 INFO:teuthology.orchestra.run.smithi187.stderr:user 25m41.856s 2023-10-12T20:03:54.504 INFO:teuthology.orchestra.run.smithi187.stderr:sys 1m5.885s 2023-10-12T20:03:54.505 INFO:tasks.ceph:Archiving logs... 2023-10-12T20:03:54.505 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584/remote/smithi170/log 2023-10-12T20:03:54.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-12T20:06:30.507 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584/remote/smithi187/log 2023-10-12T20:06:30.509 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-12T20:10:24.829 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-12T20:10:25.066 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-12T20:10:25.067 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-12T20:10:25.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-12T20:10:25.191 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-12T20:10:25.224 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-12T20:10:25.225 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T20:10:25.229 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T20:10:25.241 INFO:teuthology.orchestra.run.smithi170.stderr:bash: ntpq: command not found 2023-10-12T20:10:25.244 INFO:teuthology.orchestra.run.smithi187.stderr:bash: ntpq: command not found 2023-10-12T20:10:25.272 INFO:teuthology.orchestra.run.smithi187.stdout:210 Number of sources = 4 2023-10-12T20:10:25.272 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T20:10:25.273 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2023-10-12T20:10:25.273 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +198us[ +235us] +/- 75ms 2023-10-12T20:10:25.273 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -1158us[-1121us] +/- 46ms 2023-10-12T20:10:25.273 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +1588us[+1588us] +/- 65ms 2023-10-12T20:10:25.274 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T20:10:25.282 INFO:teuthology.orchestra.run.smithi170.stdout:210 Number of sources = 4 2023-10-12T20:10:25.282 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T20:10:25.283 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-10-12T20:10:25.283 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 +171us[ +177us] +/- 75ms 2023-10-12T20:10:25.283 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -1197us[-1191us] +/- 46ms 2023-10-12T20:10:25.283 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +1567us[+1573us] +/- 65ms 2023-10-12T20:10:25.284 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T20:10:25.285 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-12T20:10:25.296 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-12T20:10:25.297 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-12T20:10:25.322 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-12T20:10:25.524 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-12T20:10:25.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T20:10:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T20:10:25.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T20:10:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T20:10:25.758 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-12T20:10:25.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-12T20:10:25.779 INFO:teuthology.task.internal:Duration was 5885.759170 seconds 2023-10-12T20:10:25.779 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-12T20:10:25.789 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-12T20:10:25.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-12T20:10:25.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-12T20:10:25.868 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T20:10:25.871 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T20:10:26.184 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-12T20:10:26.185 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2023-10-12T20:10:26.185 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-12T20:10:26.227 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2023-10-12T20:10:26.227 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-12T20:10:26.254 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-12T20:10:26.255 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-12T20:10:26.270 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-12T20:10:26.353 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-12T20:10:26.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-12T20:10:26.356 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-12T20:10:26.437 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-12T20:10:26.451 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-12T20:10:26.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-12T20:10:26.489 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-12T20:10:26.534 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-12T20:10:26.546 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-12T20:10:26.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-12T20:10:26.599 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2023-10-12T20:10:26.608 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2023-10-12T20:10:26.622 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-12T20:10:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T20:10:26.637 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-12T20:10:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T20:10:26.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-12T20:10:26.688 INFO:teuthology.task.internal:Transferring archived files... 2023-10-12T20:10:26.689 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584/remote/smithi170 2023-10-12T20:10:26.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-12T20:10:26.725 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584/remote/smithi187 2023-10-12T20:10:26.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-12T20:10:26.763 INFO:teuthology.task.internal:Removing archive directory... 2023-10-12T20:10:26.764 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-12T20:10:26.766 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-12T20:10:26.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-12T20:10:26.825 INFO:teuthology.task.internal:Not uploading archives. 2023-10-12T20:10:26.825 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-12T20:10:26.835 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-12T20:10:26.836 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-12T20:10:26.838 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-12T20:10:26.850 INFO:teuthology.orchestra.run.smithi170.stdout: 262927 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 12 20:10 /home/ubuntu/cephtest 2023-10-12T20:10:26.865 INFO:teuthology.orchestra.run.smithi187.stdout: 262239 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 12 20:10 /home/ubuntu/cephtest 2023-10-12T20:10:26.867 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-12T20:10:26.878 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-12T20:10:26.901 INFO:teuthology.nuke:Checking targets against current locks 2023-10-12T20:10:26.930 DEBUG:teuthology.nuke:shortname: smithi170 2023-10-12T20:10:26.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-12T20:10:26.955 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-12 18:25:17.519160', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1m8XjWbqD1iXBJEhpX/v8gnKr8KzzzsWDDpPq6Zl+EF15uFoyzlAjob/fRe4yJl3EC86XUqN4NQNwEQD1PzoY='} 2023-10-12T20:10:26.959 DEBUG:teuthology.nuke:shortname: smithi187 2023-10-12T20:10:26.959 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-12T20:10:26.979 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-12_17:57:07-rgw-wip-yuri2-testing-2023-10-11-0853-pacific-distro-default-smithi/7423584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-12 18:25:17.521010', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKwWJMgAdGT2iiMDleMD67QKtvCoO64aQexf5vcn57O0419KVr8ejNnxld6r3GrUe8l2tWI+ssJ7+UJv7eJPIME='} 2023-10-12T20:10:27.015 INFO:teuthology.orchestra.console.smithi170:Power off 2023-10-12T20:10:27.017 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-12T20:10:27.044 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2023-10-12T20:10:27.045 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T20:10:27.065 INFO:teuthology.orchestra.console.smithi187:Power off 2023-10-12T20:10:27.066 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-12T20:10:27.078 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-10-12T20:10:27.091 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2023-10-12T20:10:27.092 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T20:10:27.120 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-10-12T20:10:31.079 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T20:10:31.123 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T20:10:31.191 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-10-12T20:10:31.236 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-10-12T20:10:35.194 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T20:10:35.237 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T20:10:35.309 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2023-10-12T20:10:35.350 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2023-10-12T20:10:35.411 INFO:teuthology.orchestra.console.smithi170:Power off completed 2023-10-12T20:10:35.451 INFO:teuthology.orchestra.console.smithi187:Power off completed 2023-10-12T20:10:35.558 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} duration: 5885.759169816971 failure_reason: rgw multisite, test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0882e8ebe0604dd3a24e8ead69eab402 status: fail success: false 2023-10-12T20:10:35.559 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-12T20:10:35.643 INFO:teuthology.run:FAIL